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ABSTRACT: Online environments can cultivate what have been referred to as 21st century skills 
and capabilities, as youth contribute, pursue, share, and interact around work and ideas. Such 
envi ronments also hold great potential for addressi ng digital divi des related to the development 
of such skills by connecting youth i n areas with fewer resources and opportunities to social and 
material supports for learning. However, even with increasing attention to the importance of 
21st century skills, there is still relatively little known about how to measure these sorts of 
competencies effectively. In this paper, we offer an exploratory approach for interpreting 
student user trace log data to reveal opportunities for creative production, s el f-directed learning, 
and social learning online. Our approach engages social learning analytics to code actions 
according to relationships between users and engages in self-report and ethnographic methods 
to supplement initial results. We share our methods; provide rich description of the unique 
learning environment; present results of logged opportunities for creative production, self- 
directed learning, and social learning across the sixth grade cohort; and explore these results 
through the lens of individual learners, including cohort self-reports of identity, interest, and 
perceptions, and qualitativecasestudies oftwo students. 

Keywords: Social learning analytics, K-12 learning environments, middle school, creative 
production, self-di rected learning, I earning communities 


1 INTRODUCTION 

Online environ merits for K-121 earning are escalating in both their design and their adoption into formal 
and informal education spaces (Bell et al., 2009; Allen & Seaman, 2008). At the same time, there is 
increasing attention, reflected in national policy reports and standards for K-12 learning, on the 
development of skills and dispositions necessary for productive participation in the 21st century. Exact 
definitions vary, but the re is general agreement about the importance of interpersonal capabilities, such 
as communication and collaboration, and individual capacities, such as creativity and persistence (e.g.. 
Partnership for 21st Century Skills, 2011; Pellegrino & Hilton, 2012). There is evidence to suggest that 
online environments can cultivate these sorts of skills and capabilities, as youth contribute, pursue, 
share, and interact around work and ideas (e.g., Jenkins, Purushotma, Weigel, Clinton, & Robison, 2009; 
Ito etal., 2009). Online networked environments also hold great potential for ad dressing digital divides 
related to the development of such skills by connecting youth in areas with fewer resources and 
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opportunities to social and material supports for learning (Aspen Institute, 2014; US Department of 
Education, 2010). 

Despite this situation that seems poised for impact, we still have much to learn about what works in K- 
12 online learning in terms of design features, content, and pedagogy, how to provide evidence of 
outcomes, oreven whatthose outcomes are (Means, Bakia, & Murphy, 2014). What we do know is that 
many online instantiations used in educational settings mirrortraditional learning environments and 
practices, where a teacher provides assignments and students submit work and are graded on 
completed work and time on task (McGloughlin & Lee, 2010; Bacow, Bowen, Guthrie, Lack, & Long, 
2012). We also know that development of those abilities articulated as necessary for productive 
citizenship in the 21st century demand more than traditional instructional approaches (Barron, Gomez, 
Pinkard, & Martin, 2014; Bell et al., 2009), and that evidence of these types of nuanced capacities are 
difficultto measure. 

As such, there is a growing recognition that we need a variety of methods to conceptualize and measure 
success and learning in online platforms, and that it might look different for different communities and 
contexts, even those using the same technical system (e.g., Breslow et al., 2013; Means, Bakia, & 
Murphy, 2014). In this paper, we share one approach for using learning analytics to look at evidence of 
21st century I earning activities exhibited by young learners as they engage in K-12 online environments. 
Specific research questions driving our work are these: 

How can we identify evidence of potential opportunities for 21st century learning activities i n 

logdata of usertraces? 

How can we better understand 21st century online learning activity patterns in different 

communitiesof learners with theirown practicesand intentionsof use? 

To answer these questions, we share research from one teacher and his sixth-grade students (Hand 12 
years old) as they used i Re mix, an online social learning network designed to engage youth in creating, 
sharing, and interactingaround multimedia artefacts. We engaged in learninganalyticsasa way to use 
log data sets to examine activities on the system over time at the level of individual students and a 
community of learners. Learning analytics is defined as collecting, analyzing, and reporting data about 
learner actions and interactions within their specific context, with the intention of better understanding 
and optimizing learning environments (Siemens & Long, 2011; Siemens, 2012). A sub-strand of this 
emerging and increasingly important analysis and design space is social learning analytics, which focuses 
on interactions between users to highlight potential social learning processes as an important addition 
to more traditional measures of learning activities, such as time-on-task or completion rates 
(Buckingham Shum & Ferguson, 2012). This greater attention to sociocultural dimensions of the 
individualusersinthesystemandthe learning processes between them has the potentialtoadd to the 
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"pedagogical and ethical integrity" of the analysis of user log data (Buckingham Shum & Ferguson, 
2012 ). 

In this paper, we start with theoretical and design background to position our work, includingdefinitions 
of 21st century learning, related research looking at opportunities for 21st century learning in K-12 
blended learning environments, and the learning and design framework underlyingthe iRemix platform. 
We then detail our multiple exploratory methods used to answer the primary research questions. 
Results are laid out in three parts addressing focal questions through different levels of analysis. First, 
we share a rich qualitative case of the face-to-face learning environment, including the school, the 
teacher, and the classroom over the course of the year. Second, grounded in the qualitative context, 
results of student log data analysis are presented and triangulated with self-report survey data to look at 
patterns within one unique community of learners. Third, we return to qualitative data to look deeply at 
twostudentlearnerswithinthe community who exhibited different levels of activityon the syste m to 
better understand log patterns. We close with a discussion of results, and plans for future work. 

This paper contributes to existing research in K-12 blended learning environments, and shares an 
exploratory approach for understanding activity in online learning platforms in ways that go beyond 
traditional metrics of success. We believe that, although in an early stage, this work has implications for 
researchers, designers, and educators who strive to support youth from different communities to 
participate onlinein becoming21st century learners. 

2 BACKGROUND 

2.1 Defining 21st Century Learning 

There is widespread acknowledgement both that the rise of computing is resulting in an increase in 
workforce demand across occupational fields for those who can e ngage in complex human thinking and 
communication, and that our present system of education is not prepared to teach or evaluate these 
more abstract abilities (Levy & Murnane, 2004). At the same time, there is less demand forthose who 
perform more procedural content and tool-specific tasks that can be automated. As a result, 
stakeholders in education and workforce development, including policy-makers (e.g., Pellegrino & 
HiIton, 2012; NGSS Lead States, 2013; National Governors Association, 2010) and industry heavyweights 
(e.g., Partnership for 21st Century Skills, 2011) are working to define, assess, and support development 
of aspects of 21st century learning. 

Although there is no one definition or agreed-upon taxonomy of 21st century skills and discussions of 
21st century learning attend to various components of what this means (i.e., dispositions, skills, 
orientations, and competencies), there is usually attention to both individual and interpersonal 
dimensions. The Partnership for 21st Century Skills argues that success is dependent on four broad and 
transferable skills: critical thinking, communication, collaboration, and creativity (2011). The Assessment 
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and Teaching of 21st Century Skills project, comprised of international education officials and 
representatives from leading technology companies, include four strands: 1) ways of thinking (creativity, 
critical thinking, learning to learn), 2) ways of working (communication, collaboration), 3) tools for 
working (informational and technical literacy), and 4) living in the world (citizenship, life and career, 
personal and social responsibility) (Griffin, McGaw, & Care, 2012). A National Research Council report 
(Pellegrino & Hilton, 2012) organizes 21st century skills into three strands of competencies, compri si ng 
cognitive (including problem solving and creative ability), intrapersonal (including self-management, 
intellectual openness, and work ethic), and interpersonal (includingsharing knowledgeand working with 
others). 

While these efforts have resulted in some inroads to changing standards (e.g., cross-cutting science 
practices in the Next Generation Science Standards (NGSS Lead States, 2013), longstandingsyste ms are 
difficult to change. We are interested in the potential for K-12 online environments to supplement 
traditional instruction in order to support the development of 21st century learning. Jenkins and 
colleagues' (2009) outline of necessities for engagement in informal online participatory cultures, for 
example, reflects aspects of definitions of 21st century learning outlined above, including willingness to 
engage in collaborative work, knowledge of how to manage information, self-direction of one's own 
learning, and buildingof collective intelligence. 

2.2 Opportunities for 21 st Century Learning in Online Environments 

Online education opportunities are increasingly present in the classroom, often positioned as a potential 
solution to challenges facing contemporary formal education, such as limited space, prohibitive costs, 
and issues related to accessibility and equity. A 2009 report estimated that 75% of public schools had 
one or more students enrolled in a fully online or blended course, and 66% estimated thattheir online 
offerings would grow (Allen & Seaman, 2008). In 2014, an estimated $21 billion ofthefundsspent in K- 
12 schools were earmarked fortechnology and 63% of US districts reported contracting with vendors of 
online courses (Burch &Good, 2014). 

Less is known about the impacts these opportunities have on young learners. As schools continue to 
adopt online learning practices and technologies, there is an ongoing effort to define and classify the 
variety of ways that Internet-enabled collaboration, communication, creativity, and accessible 
information are changingthe state of education (ACCI NSF, 2011). A report issued by the US Department 
of Education found a dearth of rigorous, published research studies on the effectiveness of online 
learning environments for K-12 education (Means, Bakia, & Murphy, 2014). That report includes a meta¬ 
analysis of 51 empirical studies of online learning that showed that students in online learning 
conditions performed betterthan those receivingface-to-face instruction, but caution was advocated i n 
interpreting results because of the extreme variability between platforms, ages, context, time, and 
perhaps most importantly, definitions of learning. 
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There is growing consensus that online and blended learning implementations require new strategies, 
policies, andtoolstobesuccessful, and newwaysto measure success. The National Science Foundation 
Advisory Committee for Cyberinfrastructure calls out the need for a "revolutionary" approach to 
paradigmatic shifts and systemic structural reform as well as more immediate changes to technology- 
based curriculum to meet current workforce standards (ACCI NSF, 2011). Researchers argue that 
educators must find ways to utilize technology as more than just a way to deliver curriculum, engaging it 
for critical, creative, and social purposes that leverage capabilities (Buckingham, 2007). We see an 
opportunity to reframe and supplement how we look for evidence of learning and success in formal 
environments by attending to ski I Is and practices that have been identified as cutting across different 
content areas and grade levels, those defined as part of the corpus of 21st century learning, and by 
building on what we know about learning in informal online spaces. 

2.3 Evidence of 21st Century Learning in Online Social Networks 

Though the specific features of social network sites vary, they have been defined as web-based services 
that allow individuals to construct a public or semi-public profile within a bounded system, articulate a 
list of other users with whom they share a connection, and view their list of connections and those 
made by others in the system (Boyd & Ellison, 2007). 

Existing studies of I earning and developmental affordances of common social network sites without a 
particular topic or focus beyond connecting to others, such as Facebook and MySpace, suggest 
relationships between usage and some cognitive skills, such as developing technological fl uency (Ahn, 
2013) and digital literacies (Greenhow& Robelia, 2009a), and intrapersonal and interpersonal capacities, 
such as trying out new socially responsible identities (Greenhow & Robelia, 2009b), and communicating 
with otherstudentsaround academics (Salaway, Caruso, & Nelson, 2008; Selwyn, 2009). 

Online communities can also be more focused, comprised of a blend of expert and novice participants 
who come together around a common interest or pursuit, such as those formed around massive multi - 
player online games, online forum health and support communities, and how-to maker groups. Notions 
of learning in online networked spaces build on the work of Lave and Wenger (1991), who defined a 
community of practice as a group of people with a common profession or craft, where learning occurs as 
a social process as knowledge isco-constructed by members sharing information and experiences. For 
youth, examples include the Scratch online community, an online introductory programming 
environment where kids can create, share, and remix projects, and Project Noah, an online citizen 
science site with classroom components allowing youth to upload species observations as photographs 
and GIS information and share results and get feedback. Often, these platforms incorporate social 
network features to support these peer-to-peer interactions. The also include members hip and some 
formal policiesaround usage. 
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Studies have identified youth using online communities in ways that connect with 21st century learning 
as they share their ideas and work with a community, learn from others, find learning resources, 
establish partnerships, and make choices about their future. An ethnographic study of yo ung people's 
use of digital media found that through participation in online communities or special interest groups, 
young people were learning new skills, assuming public roles as digital artists and producers, editors, 
and directors, developingimportant learning relationships, and makingsense of complex ideas (Ito et 
al., 2009). In another study, case studies of girls highly engaged in online comm unities revealed both the 
development of technological literacy and new identities such as global citizenshi p (Guzzetti, 2006). A 
study of participants in online gaming communities found individuals developing high-level collaboration 
skills, and taking new knowledge learned within the gaming community back into their real -world 
employment (Walter, 2010). Research of FI ickr participants, an online photography sharing site, showed 
individuals learning from each other and creating new meanings together for their collective 
photography practices and the normed behaviour practices on the site (Robbie & Zeeng, 2008). 

In the following section, we share the iRemix platform design, which was developed intentionally to 
capitalize on the engagement of youth in informal social learning networks and online communities. 

3 PLATFORM DESIGN: IREMIX 

Since 2004, the Digital Youth Network (DYN) has developed formal and informal I earning opportunities 
and environments for urban youth, focusing on the development of traditional and digital media 
literacies, and a related community of practice (Martin & Pi nkard, forthcoming; Barron, Gomez, Pinkard 
& Martin, 2014). DYN programs and platforms are offered in formal school classroom instruction and 
informal community spaces, such as afterschool and libraries. Building on a learning ecologies 
frame work that envisions learning as an ongoing process (Barron, 2004; 2006), DYN utilizes technology 
infrastructures to engage youth in productive learning experiences across physical spaces of home, 
school, and community. 

iRemix (see Figure 1) is a DYN-developed closed social network designed for middle- and high-school- 
aged youth (eleven-to-eighteen year olds) used across the city of Chicago in formal and informal 
environments. The site was designed to fosterthe development of youth media projects and to connect 
a community of learners, including students and educators, around these creative artefacts. Although 
iRemix can be used in different ways, with different content focus, in different sorts of e nvironme nts, 
the overarching learning goal for the platform mirrors that of the DYN program, to support the 
development of digital media citizens through production, reflection, critique, and communication 
(Barron, Gomez, Pinkard, & Martin, 2014). The purpose of the iRemix site aligns with research 
suggesting that the future of media and information is in the hands of youth who have the powerto 
create and share theirown perspectives (Gauntlett, 2012). 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs S.O Unported (CC BY-NC-ND 3.0) 


42 


JOURNAL OF LEARNING ANALYTICS 


S LAR 


(2016). Revealing opportunities for 21 st century learning: An approach to interpreting user trace log data. Journal of Learning Analytics, 3(2), 
37-87. http://dx.doi.Org/10.18608/jla.2016.32.4 



Figure 1: Amaya's home page of the iRemix site. 

The iRemix platform provides a classroom, school, or program the ability to customize a closed social 
learning network instance and to control who can access it. Each instance has its own URL, logo and 
colour scheme, and I earning activities. We define iRemix as an online social learning network, a Web- 
based environment that uses features of social network sites and learning management systems to 
support and develop an online I earning community and the individual participants within it (see Figure 
2). Within the environment, the individual learner interacts with the system and the embedded 
resources within that system (youth-system interactions) and also with other learners on the system 
(youth-adult educator interactions and youth-peer interactions). In this section, we look at how 
features of learning management systems (integrating more formal learning approaches) and social 
networks (integrating informal learning approaches) were intentionally used in iRemix to support DYN 
programmatic goals, including development of technological literacy through creative production, 
community participation through communication and critique, and self-directed learning through 
utilizing networked resources and monitoring progress. 
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Participants have 
their own profile 
page, can link to 
others, and can 
see these links. 


community of Learn ers 


Social Network 
Features 


Learning 

Management 

System 


/^dividual Learner 


Community members 
share knowledge and 
interests, and peer-to- 
peer interaction is a key 
driver of activity and 
motivation. 


Educators prompt, organize, 
scaffold, monitor, and assess 
work and assignments. 


Figure 2: Annotated representation of the multidirectional learning supports in iRemix. 


3.1 Using Learning Management Features to Support Project-Based Instruction 


Learning management systems (or course management systems) refer to computer-based systems 
designed to assist instructors and learners in the management and administration of course 
dissemination and participation, particularly distributing course content and tracking student 
performance (Ullman & Rabinowitz, 2004). Teachers can organize their lessons, upload and distribute 
content, facilitate student collaboration through forums or wikis, connect with other teachers, pare nts, 
and administrators, conduct assessments, and track and analyze classroom reporting such as grades, 
attendance, homework, etc. Examples in K-12 include adaptable open-source packages such as Google 
Classroom, EdModo, and Moodle and proprietary systems such as Blackboard Learn and Pearson 
SuccessNet. Through learning management systems, educators contribute the content that their 
students will engage with and can structure how and when that will happen, usually within a bounded 
time period, such as a course or an academic year. 

iRemix purposefully incorporates features of learning management to support project-based learning 
through artefact-driven curricula. DYN's focus on project-based learning builds on the idea that youth 
should be creators, not simply consumers of technology, and the site is structured around the 
submission of digital artefacts in multimedia formats (including images, video, audio, and text 
documents). Recent research suggests that engagement in creative production activities can help 
nurturea range of 21st centuryskillssuchasan innovative mindsetandasense of creative self-efficacy 
(Barron & Martin, 2016; Blikstein, 2013; Vossoughi & Bevan, 2014), the development of confidence with 
tools, a sense of ability in gene rating novel ideas, skills in using networked resources for learning, and a 
willingness to engage in rounds of revision and troubleshooting (Barron, Gomez, Pinkard, & Martin, 
2014; ltoetal.,2009). 

Specific learning management features that allow educators to guide the work that youth engage in on 
the platform include templates for scaffolded learning pathways, guiding educators to upload sequences 
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of assignments culminating in a digital media submission (Madison-Boyd & Steele, 2015), and 
assessment mechanisms such as rubric ratings that align to the requirements and learning dimensions 
set in the learning pathway activity formation. Within learning pathways or other one-off calls for 
student work such as through a blog post, educators can build in learning supports for students to 
enable more independent progress toward project completion, such as embedded and linked resources, 
how-to guides, and cl ear assessment dimensions. Youth learners can navigate through available learning 
pathways or one-off assignments, upload their work to the system within the calls for particular 
activities, and directly ask questions about the assigned work through messaging. Youth and educators 
can review submissions and participation by looking at individual student progress along a pathway and 
through individual portfolios of work. 

3.2 Using Social Network Features to Support Community Development 

While adult educators hold key positions in the i Re mix network as they organize, structure, and facilitate 
learning in the online space, the system is also designed to foster a learner-driven community that 
reflects youth voice and interests. Importantly, students can access the system out of class and on their 
own time. One design decision was to allow youth to post digital artefacts in various spaces on the site, 
at any time, not just as submissions to educator-driven prompts. Features all owing youth to contribute 
individual ideas and work outside of assignments include status updates, comments, blogs, debates, and 
forums. All media submissions can be identified by the user as original (created by the user) or not. 
Additionally, incorporating ways for youth to interact with each other around their work was critical as 
was support for ways for youth to connect to adult mentors and educators outside of face-to-face time, 
building on the importance of peer-to-peer and youth-mentor supports and interactions that were 
critical to the success of DYN in face-to-face settings (Barron, Gomez, Pinkard, & Martin, 2014). 

Social networkfeatures wereintentionally builtintoiRemixtocultivateacommunity of learners who 
shared an interest in and practice of media project creation and development. Users are able to create a 
profile page, I ink to peers through shared classes and friendship nominations, andviewothersonthe 
system. Individuals on the system and their work are identifiable and observable, regardless of expertise 
or age. Users can form groups and start discussions around shared interests, problems, or practices, 
including those not formally part of the program curriculum. Multiple methods for one-to-one and one- 
to-group communication are available to educators and youth on the system, including private 
messaging, commenting, debates, forums, and discussion threads. 

Below, we summarize the primary learning activities the system was designed to support and the 
corresponding! Re mix features (see Table 1). For the sake of clarity, we organize these features in terms 
of discrete intentions, but many activities and features are interconnected, and in fact the redundancy 
and interwoven nature of the site is intentional to allow multiple entry points to participation. 
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Table 1: Specific iRemix features mapped onto youth learning intentions. 

Intention Features Screenshot of feature Description 


Create and Multimedia 

share wo rkand submission 
ideas formats (text, 

video, sound, 
games, photos, 
designs) in 
multiplespaces on 
the site 


Explore Learning pathways 

learning 

opportunities Embedded and 

linked references 
and resources 


Refl ect on Portfolio 

one's own 

work Pathway progress 


Develop Profiles 

identity as a 

creator Status updates 



Production of digital media artefacts, 
revising based on review of work and 
external feedback, and submission 
online. Expression of ideas through 
multimedia and authentic audience 
sharing. Workcan be submitted within 
formal learningpathways orin response 
to adult-educator prompts for 
submission, as well as at any time in 
other spaces, such as the activity feed, 
forums, or groups. 



Seeking out ideas and resources for 
I earning and creation, incl uding those 
related to required assignments and 
opportunities forgoing broader or 
deeper in a certain area. Pathways also 
includeexemplars submitted by others, 
which can be explored to generate new 
ideas. 



!iL 


Portfolios are public, displaying all work 
submitted by one user, grouped by 
media type. 

Pathway progress visualizations show 
completion and work yet to do, 
supporting I earners in planning and 
reflecti ng. 



Each userhas a public,editableprofile 
page includingusername, image, and 
spaceto tell about themselves through 
text, images, video, and HTML 
formatting. It is linked to portfolios of 
work. 


Brief status updates can be entered 
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from the home page and are broadcast 
to the site activity feed. 


Observe and 
connect with 
others engaged 
in similarwork 


Visible links to 
other users 

Groups 

Activity feed 



There are multipleways to view 
individual participation as part of a 
larger community of practicethat 
includes adulteducators and peer 
experts and novices, with the ability to 
identify friends andform andjoin 
groups based on special interests. 
Clickingon visibleusers leads directlyto 
their profilepage and portfolioof work. 

The activity feed shows a liveupdate of 
activity on the site, includingwhoand 
what, all owing users to open individual 
profiles and/or view a related artefact. 


Communicate 
with others 
around work 
and ideas 


Critique through 
ratings 

Comments 

Private messagi ng 

Forums 



Various methods and modes to express 
opinions, critique ideas, develop 
evidence-based arguments, and widely 
broadcast ideas, both privately to 
individuals and publically to the enti re 
network. These multipleentry points for 
communication areintended to support 
learners in direct interaction with others 
on the site, specifically around workand 
ideas. Examples includeratings systems 
to a I low users to rate the projects of 
others usingrubrics orthe comment 
function. 


4 METHODS 

Goals for this exploration of user trace log data were twofold: first, to interpret raw log data in ways that 
feltauthenticforthe unique contexts of use, allowingforinterpretation by educators and DYN learning 
designers that could lead to changes in design and practice, and second, to contribute to larger 
conversations about using I earning analytics to explore the potential for 21st century learning in online 
spaces. We use a "bottom up" approach to study a sociotechnical system, looking at a set of automated 
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logged actions for a smaller set of users combined with qualitative data from the environment to bette r 
understand variation in patterns of use and intentionality about, or barriers to participation. 

4.1 Participants and Local Context 

We report on one English Language Arts teacher, Mr. Kellog, who used iRemix with his sixth grade 
students (N=53) oversix monthsfromJanuary to June 2014. Kellogtaughtin a K-8 publicschool serving 
500 students. The school was located on the far west side of Chicago, and although the school was open 
to anyone within city limits through a lottery system, it drew the majority of its students from the 
predominantly Latino local community. The student sample in this study reflected the larger school 
demographics, and included47.2%girlsand 52.8% boys, with 89% self-reportingas Latino, 6% black, 2% 
white, and 2 % Chinese. Over three-quarters of these students (83%) reported Spanish-speaking 
households. Home technology access and activities for this sample looked similar to teens generally in 
the US (Lenhart, 2015). Most (85.1%) had theirown computerat home and almostall (96.2%) re ported 
home Internet access, although only about half (48.9%) said that it worked consistently. The majority of 
students regularly (more than once a week) played digital games (77.6%), used a social network site 
(75.5%), looked for information online for homework (69.4%), and used the Internet to look for 
information related to their interests or hobbies (62.5%). 

4.2 Data Collection and Analytical Strategies 

Our first research question asks how we can identify evidence of opportunities for 21st century learning 
in log data of user traces. Our second question asks how we can understand activity patterns in different 
communities of learners with their own practices and intentions of use. In this exploratory work, our 
data collection methods drill down from the system level to the individual to better understand the 
learning environment from various perspectives, helping us to interpret log data. To answer the first 
question, we first collected iRemix user trace log data from the student cohort and engaged in learning 
analytics, applying an exploratory analytical process and framework for identifying evidence of potential 
opportunities for 21st century learning. We then used self-report student survey data to explore 
correlations. To answer our second question, grounding what we saw and continuously refining our 
interpretation of results, we also engaged in extensive qualitative case studies of the classroom, teacher, 
and youth learners. Below we detail these three data collection strategies and analytical approaches. 

4.2.1 Creating a framework of interpretation 

In addition to the data generated through general use of the iRemix platform, a separate data table of 
records of iRemix trace logs is maintained. The creation of records in the trace logs database is triggered 
by user actions and certain system events determined by the DYN technical development team to be of 
interest (e.g., posting updates to the activity feed). These have evolved along with the system and the 
organization overtime. At the time of this study, there were 158 unique trace log records. A unique log 
record is defined by the action (n=40) and the type of object involved in the actions (n=38). Examples of 
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actions include login, read (looking at work and the profiles of individuals), create (posting media or 
starting a blogpost), comment (adding a comment to posted work or a discussion), and rate (formally 
assessing posted work). Examples of types of objects connected to actions include blogpost, video, 
photo, user, and group. For example, the action-type read-photo is a unique type of logged record, and 
is generated when the user clicks an interface control (i.e., a button) in order to view a photo. The 
action-type comment-blogpost indicates the user posted a comment in response to a blog post. In 
addition to the action and type, the trace logs capture otherfields includingtimestamp, the userlD of 
the person who triggered the action, the artefactID of any specific artefact (e.g., which photo was 
viewed). Some types of logs include fields containing relevant content, such as the title of a posted 
artefactand thetextentered asa comment. 

Log data was collected in Kel log's instance of i Re mix (users included 1 educator and 53 students) from 
January 2014 through the end of the school year in June (n=25,746 user trace logs). Transformation of 
log data sets was achieved to cl ear the data of irrelevant activity (i.e., researchervisitstothe site, or log 
data that was misrepresented or duplicated at the point of capture), and various exploratory analyses 
were performed on the dataset, including Tableau visualizations and SPSS descriptive and correlational 
analysis to look for patterns. This cycle of pattern finding and data cleaning allowed us to finalize a 
dataset and prepare for further transformation and analysis. The strategy for analysis, specifical ly how 
we operationalized the raw log data to reflect clusters of log activity representing potential 
opportunities for 21st century learning includes 1) conceptual framework and rationale, 2) technical 
iRemix configuration, and 3) creating a schema of mapping raw log data onto an operationalized 
learning framework. We consider this work to align with definitions of social learning analytics 
(Buckingham Shum & Fergusson, 2012). 

Mapping iRemix logged actions onto 21st century learning themes and DYN intentions. Our first step 
was to conceptualize how the actions captured in the iRemix system map onto the broader types of 21st 
century learning activities that the DYN program and platform were designed to foster — creative 
production, self-directed learning, and social learning. Moving between the actions collected by the 
system (the data collected) and the learning intentions behind the design of the system (the intended 
feature usage based on prior DYN goals and prior work, as well as what we know about potential 
learning outcomes afforded by learning management systems and social learning networks), our team of 
researchers, with input from DYN I earning designers, technical developers, and educator practitioners, 
developed a framework to operationalize actions logged on iRemix as evidence of opportunities for 
creative production, self-directed learning, and social learning (Table 2). We consider these to be human 
capacities necessary for productive participation in the 21st century, and while they do not represent all 
areas of 21st century learning, they do align with parts of current definitions as laid out in the 
Introduction to this paper. Critically, this framework is exploratory — it is not meant to imply that 
activity on the system is evidence of learning, but rather reveals certain activity as evidence of a learning 
opportunity. This work employs the strategy of first looking for evidence of the situational opportunities 
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for learning in observed online behaviours (i.e., counting opportunities when learning was possible, not 
that it actually occurred) (Reich, Murnane, & Willete, 2012) prior to pursuing alignment of action and 
learningoutcome. 

Table 2 lays out the framework of iRemix opportunities for 21st century learning, includingthe three 
focal learning themes, 11 intended learning opportunities related to this the me, and the corresponding 
logged actions on the iRemix system. We envision these elements as the building blocks of usage of the 
system, the language that can let us look for patterns in this exploratory work. In this interpretive 
framework, careful attention is paid to iRemix as a social learning network; in other words, the meaning 
of the logged action wasdependenton the roleof both the actorand the recipient. 

The project-based focus of DYN is based on research, suggestingthat engagement in creative production 
activities, including cycles of feedback and revision, can help nurture 21st century skills such as 
creativity, technical self-efficacy, and workingthrough complexity(Blikstein, 2013; Vossoughi & Bevan, 
2014; Barron, Gomez, Pinkard, & Martin, 2014). Online profiles have been shown to promote individual 
visibility on a network (Girgensohn & Lee, 2002). As such, we classify log actions of Edit own profile 
page, Post created media (original), Edit own work, and Resubmit work as opportunities for creative 
production, as youth work to create and revise digital media artefacts and devel op their public identity 
as a creator online. 

Actions related to the learning management features in the system are intended to provide 
opportunities for youth self-directed learning, specifically allowing youth to look for new opportunities 
they are interested in, use embedded and linked digital learning resources, and seek su pport from an 
educator by asking a direct question about an activity. The learning pathway progress visualizations 
provide students with a tool to support self-regulated learning as they look at their progress and plan 
out what they have to do next (Dabbagh & Kitsantas, 2012). As such, we classify logs of View own 
pathway progress, Post question to educator about assignment, View potential activity, and Open 
activity resource asopportunitiesforself-directed learning. 

To look at opportunities for social learning, we specified those features identified as potentia I learning 
interactions with others around work, including looking at the work of individuals to get new ideas for 
their own work (Peppier, 2013), participating in interest-based communities (Jenkins etal., 2009; I to et 
al., 2009), exploring communities and individuals to see who is in the network, what they are interested 
in, and what they are doing, and actively communicating through comments on work (Berlanga et al., 
2007) or ideas, or more formal critiques (Barron, Gomez, Pinkard, & Martin, 2014). We classify log 
actions of Create new group, Join existing group, Open work of user (peer or educator), Open profile of 
user, Open portfolio of user, Open group, Open forum, Comment, and Provide critique rating as 
opportunitiesforsocial learning. 
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Table 2: iRemix21st century learning opportunities framework. 


DYN theme related to 21 st century learning 

Intended learning 
opportunity 

Logged iRemix action 

Creative production: 

Define creator identity 

Edit profilepage 

Understanding and utilizing appropriate media, 
elaboratingand refiningideas and work, creating 

Create media 

Post created media 

new and worthwhile ideas,developingmedia 
literacy and technological fluency and 
confidence through production and 

participation. 

Revise work 

Edit own work 

Resubmit work 

Self-directed learning: 

Use resources 

Open activity resource 

Reflecting on learning experiences and 
processes, personal i zing 1 earning through 

Monitor progress 

View own pathway 

makingconnections with individual interests and 


progress 

goals, taking initiative, beinga lifelonglearner, 
developing self-direction, makingdecisions, 
seeking out information. 

Seek support 

Post question to educator 
about assignment 


Seek opportunities 

View potential activity 

Social learning: 

Participateingroups 

Create new group 

Communicating and collaborating a round work 


Join existinggroup 

and ideas, being open to new ideas and 
perspectives, teachingand learningfrom others. 

Explore work of others 

Open work of user 


Explore community 

Open user profile 

Open user portfolio 

Open existinggroup 

Open existingforum 


Communicate 

Post comment 



Provide critique rating 


Streamlining technical iRemix configuration and coding process. Next, we developed a technical 
configuration of iRemix that permitted researchers to access trace log data, implemented the coding 
framework, and produced a database of coded logs that can be used for data analysis and visualization. 
Figure 3 provides a simplified overview of this configuration. A cloud-based production server provides 
storage and computing for the iRemix platform. The platform itself contains various data tables typically 
required of such social network applications, such as data related to instances, site customization 
settings, users and accounts, posted content, groups, commenting, as well as data related to more 
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specialized features such as debates, ratings, challenges, and pathways. This data is generated and 
stored as users access and interactwith the system. 



Figure 3: Technical configuration of iRemix enabling log coding. 

To serve our research and design needs in analyzing and creating visualizations based on the trace logs, 
we established a Research Data Warehouse, separate from the iRemix platform production server, to 
house the trace log data and other data not generated by the iRemix platform but needed for analysis 
and visualization (such as the 21st century learning opportunities framework). This Research Data 
Warehouse, also stored on a cloud-based server, is populated through nightly routines that copy new 
trace log data from the production server. This separation between the production server and the 
Research Data Warehouse allows for control ling access to iRemix data by instance and reduces queries 
to the production serverthat may negatively impact platform performance. 

This process of triggering the creation of trace logs, determining which fields and values are incl uded in 
the logs, and tracking the unique types of logged actions is negotiated among the iRemix programmers, 
researchers, and designers. As new features are added, or as features are altered, decisions must be 
made about how to add to or modify the creation of trace logs and how they are coded. To support 
collaboration and communication around data documentation, we created a wiki for internal staff as 
well as external stakeholders (such as administrators and researchers using their own iRemix instances). 
The wiki provides an index of key data tables linked to descriptions of fields, value types, dimension 
tables, known issues, examples, and otherannotations. 

Automating coding of raw trace logs to the 21st century learning opportunities framework. The 

database of trace logs (action-type) described earlier are the raw logs produced by iRemix. That is, they 
are meant to provide the base level information describing the actions taken. While the data provided in 
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these logs has utility for capturing user behaviour, we use coding frame works to further categorize logs. 
These frameworks re present our developing theoretical understandings as they emerge from ongoing 
research and make explicit connections to the types of actions logged in i Re mix. In this paper, we focus 
on the 21st century learning opportunities framework to better understand the online activities of 
youth. This is part of our larger work to understand interactions and learning on the iRemix site (e.g., 
coding the learning support actions of adult educators; see Nacu, Martin, Pinkard, &Gray, 2014). These 
frameworks are stored as databases to translate raw logs to coded logs in the Research Data Warehouse 
(Nacu, Martin, Schutzenhofer, & Pinkard, 2016). 

Figure 4 provides a sum mary of the coding process. For actions made by youth learners, our objective is 
to connect the raw code with an intended learning opportunity and a linked broader theme of 21st 
century learning to facilitate summary and inte rpretation. To do this, categorizing student actions fi rst 
requires codingthe logs based on the unique log type (i.e., the action-type, such as comment-video). 
Next, the coding framework examines who the action was directed towards in order to apply an 
intended learning opportunity code. That is, the coding frame work considers the actor and the recipient 
of the action. For Example lin Figure 4, if a student (the actor) clicks a button to view the posted video 
(action-type: read-video) of a peer (the recipient), this action will be coded as exploring the work of 
others, and linked to the higher level DYN I earning the me of social learning. In Example 2 in Figure 4, if 
the log data indicates that the video was created by the user (self) and the action is edit, it will be coded 
as reflecting the I earning opportunity to revise work, which relates to the theme of creative production. 


Coding method 


Example 1 


Example 2 



Figure 4: Summary of automated coding of raw trace log data. 

While iRemix provides most of the data used in our analysis through the trace logs database, other 
metadata is used to organize the trace logs. For example, while iRemix in its current configuration 
organizes user data into instances, it does not store historical classroom roster data. For research 
purposes, it is important to know how youth learners are associated with individual educators and each 
other, and howthose classroom and program rosters change from year to year. While this metadataset 
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is relatively small, it is part of the configuration that helps to organize the raw logs into the groupings 
needed for data analysis, reporting, and visualization. 

Once the raw trace logs are coded, this becomes a data source that can be used for analysis and 
visualization. A dedicated Tableau serverconnectstothe research database and allows for the creation 
of data visualizations that can be made accessible to various user groups. Tableau was selected as a 
platform for data visualization and analysis because it allows for a direct connection to the source data 
(via the Research Data Warehouse), allows for quick prototyping and iteration on interactive data 
visualizations, and provides robust use rand group controls for providing access to subsets of data. The 
coded data logs can also be exported for use i n SPSS (and othertools as needed) to be considered along 
with otherdata, such as surveys, forquantitative data analysis. 

Importantly, user trace log data may not always reflect intentional decisions to click or not click certain 
areas of the site. Thus, while we are interpreting the absence of data in user logs for a certain area as 
evidence of the user lacking the initiative to trigger that log, it may rather be that the user did not know 
howto access or use a particularfeature. Similarly, there is a chance that some logs represent parts of a 
click-through path necessary to get to another intended activity. Through our initial process of meaning 
making as we refined our frame work for 21st century learning opportunities, we tried to minimize these 
types of clicks. For instance, the log data consisted of an overwhelming number of unique log records of 
students looking at their own work on the system. On average, 39.52% of the overall corpus of students' 
trace actions (M=122.57, SD=36.67) identified an instance when the user was on a screen that portrayed 
their own posts. Although self-review is a critical element of online participation and the development 
of high-quality project-based work, this log skew led us to consider logged actions that were more likely 
driven by cognizant user choices versus those that could potentially happen as the user was on the way 
to do something else. Reports of students looking at their own work were generated whenever users 
navigated to their own profile page or a learning pathway they were working on that day. In other 
words, this log could be a result of a workflow path click-through as opposed to an intentional discrete 
activity. These actions, therefore, are not currently part of our 21st century learning opportunities 
framework. The activity logs we did include in this analysis were considered relatively dependent on 
some user choice as opposed to a common system path log, although this is not 100% accurate as users 
navigate and explore the system in their own unique ways. 

4.2.2 Correlations in online activity with measures of 21st century learning orientations 
To further explore the authenticity of using log data to identify evidence of potential opportunities for 
21st century learning, we collected self-report survey data from the student cohort (N=53). Metrics 
included those used in previous studies of youth orientations and activities related to 21st century 
learning (Barron, Walter, Martin, & Schatz, 2010; Barron, Gomez, Pinkard, & Martin, 2014; Barron & 
Martin, 2016). The survey used Li kert-response items, checklists, and rating scales cove ring themes of 
access to technology; prior experiences with digital learning; self-reported interest, confidence, and 
valuing of technology; plans for the future; and technology social learning network (see Table 3). 
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Questions focus on technology and digital media learning in particular given the content focus of DYN 
programming. Surveys were administered near the end of the school year (May 2014) during class time, 
and took approximately 20 minutes (ranging from 14-27) to complete. Importantly, these surveys were 
not meant to imply that youth had developed certain dispositions as a result of their online activities, or 
vice-versa; they were rather to explore potential correlations between our log data frame work analysis 
and exploratory self-report metrics used in priorstudies. 


Home access 


Regularonline 

hobbies 


Creative self- 
efficacy 


Confidence 
with technology 
activiti es 

Possible 
technology- 
related jobs for 
the future 

Navigatingthe 

Internet 


Sustained 
experience with 
creative 
production 
activities 


Table 3: Survey measures. 

As a proxy for socioeconomic status the survey i ncl udes questions about home access to 
technology — includinghavingtheir own computer (yes/no) and a working Internet 
connection — and a checklistformat question askingstudents to indicate their access to 
specifictools athome: wireless Internet, a printer, a video camera, a digital camera,a 
handheld mobile device likean iPad,a smartphone, and a game console. A tools at home 
measure was computed by summing the number of computers and the number of other 
tools indicated with a total possible score of 7. 

Students i ndicated how often they partici pa ted i n four online activities (garni ng, social 
networking, finding information for homework, finding information abouta hobby) on an 8- 
point scalefrom never to several times a day, and computed a binary measureto indicate 
regular activities asthosethey did more than once a week. 

A measure of creative self-efficacy uses the average of two rati ngs of agreement (/ am able 
to come up with new and interesting ideas and / am able to be creative) on a 5-point Likert 
scalefrom 1 (strongly disagree) to 5 (strongly agree); Chronbach's alpha measureof scale 
rel i a bi lity=.81. 

A measure of technology confidence uses the average of two ratings of agreement (/ am able 
to do well in activities that involve technology and / am able to learn new technologies) on a 
5-pointLikert scalefrom 1 (strongly disagree)to 5 (strongly agree); Chronbach's alpha 
measure of scale rel iability=.78. 

Students were asked to rate thei r agreement about the likel ihood of possible future 
occupations related to technology on a 5-point Likert scalefrom 1 (not at all Iikely) to5 
(extremely likely). 


Student Internet proficiency was determined based on a survey proxy to measure both 
effectiveness and efficiency by takingthe total of ratings of understandingof 15 Internet 
terms from 1 (no understanding) to 5 (full understanding), developed and validated to look 
at on line search skills (Hargittai, 2005; Hargittai & Hsieh, 2012). 

Ni ne technology activities (i ncl uding programmi ng, creating publ i cations, creati ng websites, 
making digital art, robotics, digital movie maki ng, game creation, music production, and 
developing podcasts) were presented as descriptionsof possibleproducts,such as Created a 
piece of art using an application and Created your own newsgroup blog, or discussion site on 
the Internet. Students were asked to indicate the number of times they had participated in 
each from a four-option choiceof 1) never, 2) once or twice, 3) three to sixtimes, or 4) more 
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than sixtimes. A breadth scorewas calculated representingthe number of activities a 
student participated in atleastonce and a depth scorewas calculated representingthe 
number of activities a student partici pa ted in more than sixtimes, a measure of repeated 
and sustained engagement across media genres (Barron, 2006; Barron, WaIter, Martin,& 
Schatz, 2010; Barron,Gomez, Pinkard,& Martin, 2014). 

Technology To find out more about a student's network of people they were I earning from and tea chi ng 

learning about technology, we developed counts of teaching and learningnetworks. Students were 

network asked to i ndi cate who they were teaching about computing from a list of eight relational 

categories: mother, father, sibl ings, other relatives, friends, teachers, other peers, and other 
adults. We also asked students to indicate who they learned from about computers and 
technology, from a 3-option choice of not important, somewhat i mportant, and very 
i mportant, from a choice of their teacher, other teachers atschool,adultsinan after school 
program or community centre, mother, father, sibl ings, friends, other relatives, other adults, 
and other peers. We total led the number of categories of people students reported learning 
from a lot. 


Learningfrom A question asked students about the importanceof different learningsources,froma 3- 
online option choiceof not important, somewhat important, and very important. We created a 

communities binary measure to deter mi ne which students reported I earning a lot from onl ine 
communities. 


Social 

perspectives 

online 


Students were asked to agree on a 5-point Likert scalefrom 1 (strongly disagree)to 5 
(strongly agree) with statements about their useof the Internet, includingan item I feel I've 
gotten new perspectives on societal issues (issues important to society like poverty, violence, 
health and safety, civil rights) because of my online activities (Kahne, Middaugh, Lee, & 
Feezell, 2012). We created a binary measureto determine which students reported 
agreement or strong agreement with this statement. 


4.2.3 Developing qualitative portraits of educator and focal-case learners 

To answer our second question, looking to complement and deepen our understanding of our fi rst two 
analyses, we developed qualitative case portraits of individuals (Barron, Gomez, Pinkard, & Martin, 
2014). This need to take into account the contextual setting has been identified as critical for 
understandingusertrace data in big data (Boyd & Crawford, 2012). Although ourworkdoes not classify 
as big data, this strategy encourages retaining the authenticity of unique environments of use. To 
specifically learn more about the individual intentions, goals, experiences, and identities of individual 
learners within the network, we engaged in interviews with Ke Hog and a subset of his students (n=8). 
Students were selected to represent a mix of boys and girls and a range of iRemix engagement, as 
determined by survey responses about enjoyment (more and less) and use outside of required 
participation (higher and lower). Interview protocols were adapted from existing instruments used in 
prior work (Barron, 2006; Barron, Gomez, Pinkard, & Martin, 2014), including a learning ecologies 
interview designed to elicit activities across the contexts of home, school, and community, attendi ng to 
how the youth used computers and technology in these contexts, and an artefact-based reflection 
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interview using the student iRemix profile and portfolio as a shared reference point to discuss their 
work, contributions to the site, and plans for learning more. Two teacher interview protocols were 
developed to reflect on outside interests and activities, professional goals, use of technology in the 
classroom, and student participation during the year. Interviews we re conducted at the beginning and 
end of the school year. Cases use pseudonyms throughout. 

5 RESULTS 

We report results in three sections, going back and forth between qualitative understanding of the 
unique classroom context, community and individual learners, and quantitative log data analysis. In the 
first section we provide the story of the learning environment and our focal teacher, in the second we 
present results of log coding and exploratory correlational analysis with survey data, and in the third we 
locate individual students in the log data and provide a person-centred view of their experience. 

5.1 Learning Environment: Supporting Creative Expression through Digital 

Production 

Mr. Kel log: I think everything is gearing towards ... just having a wide range of being able to take 
a simple writing piece or whatever and create a really cool media project in a variety of different 
ways, through iMovie or stop motion or ... PowToons and Garage Band and all that, so [my 
students will] go into the high school and be I ike, a 11 right, I'm going to do this project, and jus t 
wowi ng everybody. 

In this section, we describe the learni ngenvironment and the specificfactors influencing student use of 
iRemix, including the wider school culture and support for technology in the classroom and the teacher's 
goalsand priorexperiences. 

The school principal and charter administration were very involved with and supportive of technology 
for learning. In 2012, the school was part of an initiative to connect Chicago Public Schools in 
underserved neighbourhoods to the US government's Broadband Technology Opportunity Program (B- 
TOP). As part of a related grant, the school partnered with DYN and received DYN mentor support in 
middle school English Language Arts (ELA) classrooms and an Apple laptop cart. In his first year of 
teaching, the year prior to our research efforts, Mr. Kel log was selected by the administration to be one 
of the school educators involved in this collaboration. A DYN mentorvisited his sixth grade classes once 
a week supporting students to translate the written work they were doing into multimedia artefacts 
such as podcasts, videos, and graphics, blendingtraditional literacy withdigitalmedialiteracy (Barron, 
Gomez, Pinkard, & Martin, 2014). The year following the B-TOP grant, the principal chose to continue 
the partnership with DYN and DYN offered summertechnology programs atthe school and an e-fashion 
afterschool club for girls. The school also added to their own technology offerings including a middle 
school digital game design elective and a video production afterschool club. The school had a dedicated 
computer lab, classroom computer clusters, and a regularly updated website that frequently showcased 
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student media projects. The school administrators promoted data-driven instruction by utilizing a 
school-wide learning management system that collected, reported, and visualized grades and 
attendance. School professional development sessions encouraged teacher experimentation with 
technology tools in their classroom, and introduced new tools, including EdModo, an online platform for 
teachers often used to assign homework and schedule quizzes. 

At the time of this research, Kellog was in his second year of teaching sixth grade ELAand social studies. 
Despite being a young teacher in his late twenties, he did not use technology much outside of school, 
did not have a Facebook account or any other type of social media, and before his work with DYN had 
no experience with creative digital production beyond word processing. Through the collaboration with 
DYN the year prior to our research, Kellog became engaged in the use of technology for creative 
production in the classroom. Fie reflected on his own learning in this area, "Sometimes I'llbeupthere 
and, man, [I think], 'Am I really saying this right now?' Talking about i Movies and stuff, so it's cool." Fie 
and his students were recognized for their digital production expertise within the school, as Kellog 
invited teachers and administrators to class presentations and shared exemplary student work thro ugh 
emails and posts to the school website. After his first year of mandatory collaboration with DYN through 
the B-TOP program, Kellog opted to continue using DYN projects and the i Re mix platform with his ELA 
classes, this time leading the work himself with smaller collaborations with DYN to add additional 
learning pathways and resources linked to his curriculum. 

5.1.1 Goals and intentions as a teacher 

Kellog felt a responsibility toward his students that went beyond making sure they were proficient in 
ELA standards. Fie wanted to make them "ready for the next step... the next step is 7th grade, but we 
look beyond that to high school and college" and to "inspire" students to be the "best student you can 
be." Part of this was to encourage his students to take more control of their own learning, and he 
described his teaching process as one of "gradual release" from direct instruction to letting students 
take increasing ownership of their tasks. When asked specifically what he wanted his students to learn, 
he replied, 

I feel like they got iMovie down, which is great, just maybe taking it to a next level or even 
incorporating some different thin king ski I Is that go i nto designing,justtaki ng an event piece and 
creati ng something real ly cool... be more creative with it, which we'd have to real I y think outside 
the box for it. 

The response reveals Kellog's focus on supporting creative expression through the production of digital 
media, building on the initial work he did with DYN. As such, digital media in his classroom was not often 
tied to formal grades and assessments, and was positioned as something the entire class, including the 
teacher, wasexploringtogether, 
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I remember our first iMovie. It was just like we'll throw in a text box and write something on it. 

Then it got more advanced, but I think the very first time we do any of this it's going to bejust 
trial and error. Once they get good at it, that's when we startnarrowingdown expectations. 

In addition to wanting his students to learn the technical skills necessary for contemporary digi tal 
production and explore their creativity through multiple modes of expression, Kellog had a 
communicative goal forthe learningand developmentthat wenton in his classroom that was part of his 
desire to invigorate the contributive voice of his focal students: 

The culture here is they're very reserved, which allows for easier classroom management, I 
would say, but a different type of style. You really have to work to get kids to participate and 
raise their hands ... most of the challenge is getting them comfortable enough to participate and 
speak i n front of the class and share their ideas. 

5.1.2 Goals and intentions for using iRemix 

Kellog chose to continue using iRemix with his students despite the fact that, like many teachers, he and 
his students were navigating multiple platforms for management and instruction in his daily teaching. 
The school student data management system was mandatory and Kellog used it to track student 
attendance and input formal progress, including grades and test scores. He also used EdModo to direct 
assignments and quizzes to students as one-time events, and allow students to input queries of 
clarification about content or assignments. While these two systems were used for relatively traditional 
learning management and instructional practices, Kellog saw iRemix as a space for students to explore 
and share creative work and ideas. He purposefully did not utilize the more formal learning 
management aspects of iRemix like assessment of student work. Partly, this was a result of a perceived 
redundancy, since he had to enter grades in mandatory systems he did not see a reason to also do it in 
iRemix, but more importantly, he recognized iRemix as an online learning space that his students were 
excited to visit, and he wanted to keep it that way. 

A lot of these kids don't have Facebook or whatever, so I think they're all I ike, they can't wait to 
get it, this is their opportunity to try it out ... They have a profile, portfolio. They can see each 
other's work. They can see a whole history of videos, things like that... They love going on there 
and checking each other's work out, especially if it has to do with photos ... If I say, 'You have 
free time to go on iRemix," they get excited for that. They feel likeifs more personal. 

Kellog recognized the system as clearly aligned with his interest in encouraging expression through 
media artefacts. He appreciated that the system could accept and display different types of media, and 
that there was a private writingspace forcyclesof editingand revision. He was particularly interested in 
student audience as a precursor to constructive criticism and encouragement, which he thought might 
be easier for the students in his classroom to do digitally. 
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If they're doing a blog, they can post something and then somebody can read that right there 
online, as opposed to getting up from their desk and then having to look at their notebook, what 
they're writing. It's a lot easier to see and edit peers' papers and just share ideas. You can 
comment on videos. If you log in there and a student does a video, another student doesn't have 
to go walk over to that computer screen. They can stay logged in and can say, "Wow, I real ly I ike 
your video because of this." It takes away that face-to-face i nteraction, but it a I lows kids to see 
their work, get feedback. 

5.1.3 Classroom implementation of iRemix 

Du ring our study, Kel log used i Re mix approximately 43% of the days students were in the classroom (49 
days over the six-month period). His i Re mix instance included seven possible learning pathways. Kellog 
co-developedfive of them with DYN staff to align with his curriculum (e.g., see Figure 5), and the other 
two were developed by DYN: 1) Introduction to i Re mix, designed to get users familiar with the system, 
and 2) a pathwayfocusingon stop-motion animation. 

In January, students were formally introduced to the laptops, logged into i Re mix with their individual 
usernames and passwords, and started the Introduction to i Re mix learning pathway. Kellog'sfive other 
pathways were assigned to students over the next six months. The submissions required for completion 
of these pathways totalled nine media artefacts: three blog posts and two each of videos, images, and 
audio files. Kellog also posted four assignments as blog posts asking students to submit their work in 
various ways on the site; submission formats included an online private notebook post, a media post, a 
blog post, and a comment. 



Figure 5: Teacher-developed iRemix learning pathway: RACE Writing Strategy, focusing on the 
school's writing strategy for standardized testing. 

Du ring days that iRemix was a focal activity, Kel log gave his students time in class to go onto the system 
and look at required learning pathways, work on activities, and submit assignments. Students could also 
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log onto i Re mix at home on their own time. The work assigned by Kellogon the system was required, 
but the work was not formally graded. Instead, he monitored the classroom during iRemix work and 
presentation days to ensure that everyone had posted. During iRemix work time, and also classroom 
free time, he encouraged students to look at the posted work of classmates and comment on it. Twice, 
Kellog also required students to post a comment on the work of another student. He occasionally 
offered extra credit for original posts beyond requirements, but did not specifically encourage students 
to begin other learning pathways on iRemix or to review their progress using pathways and portfolio 
views. 

Kellog sometimes blended the digital and classroom I earning spaces by projecting the work submitted 
online on a screen in the physical classroom to showcase different projects and have students present 
their submitted iRemix work in person. This was linked to his intention of buildi ng student confidence 
through modes of public contribution and communication: 

I also think that community-building within Remix is good, but a lot of what we have troubles 
with, especially here, is eye contact, face-to-face interaction,speakingloudly,speakinginfront of 
the class, presenting, which we went over. A lot of the projects are, they write this great piece on 
Remix and then they upload it, but I want them to speakin front of the classroom. 

This rich description of Mr. Kellog's intentions and classroom practices allows an additional lens with 
which to interpret quantitative results. In the next analysis section, we present the automated log data 
collected from the 53 sixth graders working in this I earning environment over the course of six months. 

5.2 Student Learners: Evidence of Opportunities in Student Traces 

The students used the system an average of 52 days (SD=9.34), with individual students rangingfrom 37 
to 89 days, during the six-month study. They logged into the system an average of 70.57 times (ranging 
from 41 to 129, SD=19.48) and initiated an average of 313.26 logged user trace actions (ranging from 
130 to 488, SD=91.91). This variability suggested some level of student-initiated use of the system as 
opposed to uniform classroom participation. In this section, we apply our framework of 21st century 
learning opportunities to these logged student usertraces (see Table 2). 

Table 4 shows the average counts, and minimum and maximum counts for individual students, of the 11 
learning opportunities we were looking for, as calculated from automated coding of the individual user 
trace log actions in the iRemix system for each student over the course of the study. There was a high 
frequency of activity around youth-created artefacts, including posts of original media and students 
looking at the posts of others on the system. There we re also frequent counts of explorations of people 
and activity opportunities on the system, and evidence of communication with others. 
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Table 4: Descriptive statistics of opportunitiesfor21st century learning, as coded from usertrace log 

data. 

Activities logged for learning 


opportunity 

M 

SE 

Median 

SO 

Min 

Max 

Create media 

39.25 

1.40 

38 

10.22 

21 

62 

Explore work of others 

35.98 

2.25 

35 

16.42 

10 

101 

Seek potential activities 

26.70 

0.99 

26 

07.18 

15 

48 

Explore community 

25.53 

3.74 

13 

27.19 

0 

130 

Communicate 

22.22 

1.48 

21 

10.80 

8 

65 

Define creator identity 

21.62 

1.67 

20 

12.15 

5 

64 

Revise work 

16.96 

1.06 

16 

7.74 

2 

35 

Participateingroups 

1.87 

0.30 

1 

2.18 

0 

10 

Use resources 

1.87 

0.19 

2 

1.37 

0 

8 

Monitor progress 

0.26 

0.11 

0 

0.76 

0 

4 

Seek support 

0.13 

0.05 

0 

0.39 

0 

2 


Table 5 groups the average counts, and minimum and maximum countsforindividual students, intothe 
three focal 21st century learning themes we focus on in this paper. 

Table 5: Descriptive statistics of opportunities for 21st century learning, as coded from usertrace log 

data, grouped by theme. 


21 st Centurylearning themes 

M 

SE 

Median 

SO 

Min 

Max 

Creative production 

77.41 

3.29 

74 

23.97 

36 

135 

Social learning 

66.38 

3.67 

61 

26.71 

27 

153 

Self-directed learning 

28.96 

1.13 

28 

08.22 

15 

55 


These patterns follow what we know about the learning environment. Kellog focused on digital 
production, and youth were primarily prompted to use iRemix for creation and submission of digital 
artefacts. The educatordescribed his students as more comfortable exploringas opposed to engaging in 
direct critique or interaction, and exploring the work of others was the next most common activity on 
the system. While students were not prompted to monitor their own progress or use the systemforself- 
directed inquiry, they were encouraged to post comments and develop their own profile page; both of 
these activities were required at least once. These average trends of more and less frequent types of 
activities follow the educator-driven aspect of the formal school classroom, but there was a lot of 
variability across the individual students. The histograms in Figure 6 show the di stribution for each 
learning opportunity theme (note that the x-axis gradients are different for self-directed learning as a 
result of lower counts for that type of activity). While the creative production counts were more or less 
distributed along a normal curve, the self-directed activity and social learning activity we re skewed left, 
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indicating a higher level of variability on the higher end of the spectrum with a longer tail of highly active 
outliers. 



Creative production actions Self-directed learning actions Social learning actions 

Figure 6: Histograms of counts of clustered 21st century learning actions. 

The number of overall counts of logged activities was correlated with the number of days students used 
the system, as was the count of creative production actions, r(53)=0.75, p<.01, and social learning 
actions, r(53)=0.57, p<.01, but there was no correlation between days on the system and the number of 
self-directed learning actions. Because self-directed activities such as using embedded resources and 
monitoring progress were possible but not explicitly prompted by the classroom educator, it makes 
sense that these counts were less likely to be a function of time on the system, and differences may 
suggest more independent student choice. 

Given the variability of counts indicating opportunities for each 21st century I earningthe me, we were 
interested in finding out more about the students who evidenced more or fewer opportunities within 
each theme compared to their peers in the same I earning environment. We used a median split of the 
counts (Table 4) to create higher and lower groups for each 21st century I earning the me. Overall, 28.3% 
of students were in the lower group for all three themes, 28.3% were in the higher group for all three 
themes, and 44.4% were higher in some groups and lower in others. To explore these logged user trace 
action counts as an indicator of potential opportunity for 21st century learning, in the next section we 
use these groups to look for correlations with survey self-report data related to these themes. 

5.3 Student Learners: Comparing Log Data Patterns and Self-Report Survey Data 

For each of the three 21st century learning themes — creative production, self-directed learning, and 
social learning — we first compare students in the higher and lower groups in terms of gender, home 
access to computing and technology, and regular technology-related online hobbies. A primary focus of 
DYN work is designing for equity, and we were interested to see if any of the dimensions shown to be 
predictors of and barriers to productive contribution and participation in some computing and online 
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activities (e.g., Hargittai & Hinnant, 2008), were factors in counts of online activity that we think are 
generative for learn ingin i Re mix. Then, we compare the students in the higher and lower groups along 
sets of survey items that have been used as metrics of student 21st century I earning orientations within 
similarthemes(Barron,Gomez, Pinkard, & Martin, 2014). 

5.3.1 Creative production groups 

The lower group had creative production opportunity counts ranging from 36 to 74 and accounted for 
47.1% of learners (n=25) while the higher group (n=28) ranged in counts from 75 to 135. These two 
groups looked similar in terms of their average proportions of user trace logs that accounted for 
opportunities of creative production (43.3% and 46.2%, respectively) and social I earning activities (38% 
for both), butthe lowercreative production group had a significantly greaterproportion of self-directed 
learning activities (19.1% compared to 15.9%), (t(l,51)=2.52, p=.015). The students who were more 
engaged with production activities on the site spent less ti me focused on the aspects of the site where 
educators had set up resources for learning (i.e., progress along learning pathways, formal resources, 
and direct questions to the teacher). This may suggest that those students engaging in more creative 
production were using the site on their own time for more informal learning beyond w hat was required. 

Gender, Access, and Digital Hobbies. The two groups did not differ in terms of gender (61.5% of females 
and 44.4% of males were in the higher group), home access to computers and technology, and their 
regular online activities. However, although the groups reported spending similar amounts of ti me on 
social networking activities overall (surfing, posting, and commenting), those inthe higher group were 
significantly less likely to report surfing (70.8% of the lower creative production grou p compared to 
41.7% of the higher group, 02=4.15, p=.040). This finding potentially identifies those students in the 
higher creative production group as less likely to engage in passive exploration (i.e., surfing) in other 
networked environments, instead choosing to use theirti me to contribute. 

Our definition of creative production involves developing an identity as a creator and understanding and 
utilizing appropriate media for production (Barron, Gomez, Pinkard, & Martin, 2014). Here, we compare 
the higher and lowercreative production groups on survey measures related to creative self-efficacy, 
confidence with technology activities, and interest in taking on a future role with technology (see Table 
3). 

Creativity and Technological Confidence . The sample overall had a very high rating agreement of their 
creativity (M=4.34, SD=0.62, ranging from 3 to 5), and there were no significant differences found 
between the higher and lower groups (see Table 5). The sample also had a high rating of overall 
confidence with technology (M=4.42, SD=0.64, ranging from 2.5 to 5), and for this measure there was an 
effect of creative production group, with the students in the higher group rating higher agreement with 
this construct. 
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Table 5: Ratings of agreement of measures of creative self-efficacy and confidence with technology, by 

higherand lowercreative production groups. 


Ratings of agreement 

Group 

M 

SE 

F 

P 

Creative self-efficacy 

Lower 

4.32 

0.13 

00.04 

NS 


Higher 

4.35 

0.13 



Confidence with technology activities 

Lower 

4.11 

0.12 

13.71 

.001 


Higher 

4.72 

0.11 




Whereas all students in Kellog's class were engaged in production through poetry, writi ng, and digital 
media, and we know the teacher fostered a creative expressive environment, those students 
contributing more online were submitting digital media artefacts, perhaps accounting for the higher 
technological confidence scores for those in the higher creative production group. 

Potential Technology Careers. Those students doing more creative production activity on the system 
were significantly more likely than those in the lower creative production group to express a strong 
desire to enter fields of engineering, technologically enhanced design, and video production (see Table 
6). Again, the digital creation identity of those in the creative producer group is apparent. 


Table 6: Percentage of students who rated themselves extremely likely to go into different 

technological future jobs. 

Creative production group 


Possible technology-related jobs for the future 

All 

Lower 

Higher 

x 2 

P 

Video editor or producer 

33.3% 

17.4% 

48.0% 

5.05 

.025 

Game designer 

18.8% 

13.0% 

24.0% 

0.94 

NS 

Music producer or creator 

16.7% 

13.0% 

20.0% 

0.42 

NS 

Graphicorweb designer 

14.6% 

0.0% 

28.0% 

7.54 

.007 

Engineer or computer programmer 

12.5% 

0.0% 

24.0% 

6.31 

.014 

Network technician 

10.4% 

8.7% 

12.0% 

0.14 

NS 

Computer teacher 

6.3% 

4.3% 

8.0% 

0.27 

NS 

Writer or journalist 

6.3% 

4.3% 

8.0% 

0.27 

NS 


In sum, those students choosing to do more activities related to creative production on the system, 
including posting and revising work, had higher confide nee about their ability to work with technology 
and were more likely to think about a career in areas related to production with technology, including 
video editing, web and graphic design, and engineering. 
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5.3.2 Self-Directed Learning Groups 

The lower group accounted for 54.7% of learners (n=29) and had self-directed learning opportunity 
counts that ranged from 15 to 28, while the higher group (n=24) ranged in counts from 29 to 55. The 
groups were similar in terms of their average proportion logged as evidence of opportunities in the 
different areas of creative production (45.2% and 44.5%, respectively), social learning (38.5% and 
36.8%), and self-directed I earning opportunities (16.3% and 18.8%). 

Gender, Access, and Digital Hobbies . There were no gender differences between groups, with 42% of 
females in the higher group and 48.1% of males. There were also no differences found between groups 
in terms of home access to computers and technology, nor for frequency of time spent on online 
hobbies, includingsocial networkingandgaming. 

We define self-directed learning as seeking out information, being a lifelong learner, and making 
decisions (Barron, Gomez, Pinkard, & Martin, 2014). As such, we compare the higher and lowerself- 
directed groups in terms of their Internet proficiency, given that it is becoming the primary means of 
consuming and searching for information, and theirsustained experience with technology activities, 
suggestingtroubleshootingand persistence (see Table 3). 

Internet Proficiency . Inthissample, thegroupwith moreevidence of self-directed learning activities on 
iRemix had significantly higher scores of Internet proficiency (M=50.94, SE=2.77) compared to those 
with fewer (M=43.35, SE=2.30), F(l,42)=4.46, p=.041). 

Sustained Fluency-Building Experiences . The breadth of fluency building experience looked similar for all 
students in the cohort (M=6.62, SE=0.19) and there were nodifferences between the higherand lower 
groups. Flowever, students who exhibited more self-directed learning activities had significantly higher 
depth scores (see Table 7), suggesting persistence in learning more and finding opportunities to do so, 
despite complex multi-faceted work. We also grouped the activities into three genres — publication 
(publication, website, podcast), expressive (art, music, video), and computational (game design, 
robotics, programming) — to see which genres of production students repeatedly engaged in. This genre 
classification system is only one way to group the activities, and most span multiple categories, but it 
has been used in previous work as a way to look at patterns of participation (Barron & Martin, 2016). 
Students in the higher self-directed learning group had higher scores in each mode of technological 
production, suggesting both exploration and depth of involvement. 
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Table 7: Sustained experience with creative production activities of students with lower or higher 

evidence of self-directed learning. 



Self-directed 
learning group 

M 

SE 

F 

P 

Breadth score 

Lower 

6.63 

0.25 

00.01 

NS 


Higher 

6.60 

0.29 



Depth score 

Lower 

1.89 

0.31 

17.96 

.000 


Higher 

3.86 

0.35 



Computational depth 

Lower 

0.30 

0.12 

7.59 

.008 


Higher 

0.81 

0.14 



Publication depth 

Lower 

0.19 

0.11 

5.33 

.026 


Higher 

0.57 

0.13 



Expressivedepth 

Lower 

1.41 

0.18 

13.87 

.000 


Higher 

2.48 

0.21 




In summary, those students who exhibited more self-directed I earning activity on i Re mix, including using 
resourcesand materials, lookingforunassigned activities, and monitoring progress, had higherlnternet 
proficiency proxy scores and were more likely to have sustained experience in each type of computing 
activity we asked about, including expressive, computational, and publication. These students were 
savvy navigators of the dense and complex web of information, perhaps finding information and 
opportunities for learning more. Their depth of experience with multi media fluency building activities 
suggests persistence with particular genres of production beyond playing around (Ito et al., 2009). 
Activitiesthat can encourage development of computational fluencies have increasingly been called out 
as especially rare but important for young people (Wing, 2006, 2011; Barron & Martin, 2016). This is an 
especially interesting group to look at in the context of Mr. Kellog's classes, as we know that actions 
mapped to self-directed learning opportunities in our frame work we re not guided by the instructor. 

5.3.3 Social Learning Groups 

The lower activity group accounted for 52.8% of learners (n=28) and had social learning opportunity 
counts that ranged from 27 to 61, while the higher group (n=25) ranged in counts from 62 to 153. The 
average proportion of activities within themes was significantly different between thetwogroupsfor 
each of the three 21st century I earning themes. On average, the higher social I earning group evidenced 
a higher proportion of actions logged overall that were classified as social I earning opportunities (t(51)=- 
4.82, p<.001), and lower proportions of actions classified as opportunities for creative production 
(t(51)=2.12, p=.039) and self-directed learning(t(51)=3.24, p=.002). On average, the highergroup logged 
actions that classified as 43.1% creative production, 41.6% social learning, and 15.3% self-directed 
learning, comparedwiththelowergroup46.5%, 34.2%, and 19.3%, respectively. This highergroup may 
reflect student choice to interact with others both publically (e.g., commenting on work) and privately 
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(e.g., looking at the work of others), as opposed to those that only carried out actions in relation to 
peers or the teacher that were part of required assignments on the system. 

Gender, Access, and Digital Hobbies. While there were no significant differences found between higher 
and lower social learning groups in terms of home access to computers and technology, or in terms of 
frequency of digital hobbies, there was a significant gender difference (132=6.80, p=.009); the higher 
group was comprised of 68.0% female and 32.0% male, while the lowergroup was 32.1% female and 
67.9% male. Sixty-five percent of females were in the higher group compared with 29.6% of males. 
Interestingly, the re we re no gender differences in terms of student self-reports of use of online social 
network sites. 

We define social I earning activities as being those related to communicating and collaborating around 
work and ideas, including teaching and learning from others and being open to new ideas and 
perspectives (Barron, Gomez, Pinkard, & Martin, 2014). Below, we compare the higher and lower social 
learning groups in terms of their reports of their technology learning network, their use of online 
communities for learning, and their perceptions of getting new perspectives from being online (see 
Table 3). 

Face-to-Face Learning Networks . Both groups were teaching multiple types of people in their network 
about technology (highergroup M=3.20, SE=0.48; lowergroup M=3.18, SE=0.44). Flowever, there was a 
main effect of group for the types of people from whom they learned (F(l,46)=5.33, p=.026). Counter 
intuitively, those youth in the higher social learning group reported a significantly smaller network of 
people from whom they learned (M=6.27, SE=0.53) compared to the lowergroup (M=7.92, SE=0.48). 

LearningOnline. Overathird of students in both groups reported learning a lot from online communities 
(40.7% of the lowergroup and 45.5% of the higher), and there was no difference between groups. On 
average, students agreed that they got new perspectives on societal issues using the Internet (M=3.92, 
SE=0.17), but students in the higher social learning activity group agreed more (M=4.36, SE=0.25) than 
those in the lowergroup (M=3.48, SE=0.24), (F(l,45)=6.49, p=.014). 

In summary, the students who were higher on social learning reported learning from fewer people 
about computers and technology in their face-to-face world. They we re more likely to agree that they 
had gotten new perspectives on societal issues because of their use of the Internet. The two groups 
looked similar to each other in terms of who they taught and how they used online communities for 
learning. Kellog's class was engaging as a group in digital media skills, and it is no surprise that all of the 
students report teaching their face-to-face network, including parents, grandparents, peers, and 
siblings. Although surprisingthatthe highersocial learning group report learningfrom fewertypes of 
people, we wonder if those youth were not connected to expertise in their face-to-face learning 
network or if they may be more technologically savvy and therefore not seeking out help from others i n 
the content area of computers and technology. 
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5.3.4 Survey items across groups 

As a way of adjusting for the possibility that higher and lower counts of activity overall account for 
differences in the survey data, we compared higherand lowergroupsforotherthemesforall survey 
items. Three items — technological confidence, Internet proficiency, and depth of experience with 
technological fluency-buildingactivities — revealed significant differences between higher and lower 
groups outside of the targeted 21st century theme. 

Like the creative production groups, the higher self-directed learning group had higher technological 
confidence (M=4.69, SE=0.09) than the lower group (M=4.22, SE=0.14), F(l,46)=7.05, p=.011. They also 
were more likely to see a future in game design (38.1% compared to 3.7% of the lower group, 132=9.17, 
p=.003) and video production (57.1% compared to 14.8% of the lower group, @2=9.52, p=.003). We 
know that the learning goals and production work assigned to these students were focused on digital 
media and technology activities, so it is not surprising that those students who logged more 
opportunities for self-directed learning in the iRemix system considered themselves to be 
technologically confident and interested in digital and computational pursuits. 

Like the self-directed I earning groups, the higher social I earning group had significantly higher scores of 
Internet proficiency (M=50.90, SE=2.60) compared to those in the lower group (M=42.75, SE=2.37), 
(F(l,42)=5.36, p=.026), suggesting that those students who engaged in more social interactions online 
were developing more Internet savvy skills in general as they navigated online spaces and networked 
communities. 

Perhaps most interesting, the highergroupsof each 21st century learningtheme — creative production, 
self-directed learning, and social learning — had a higher depth of experience with fluency building 
activities than their lower group counterparts. While the self-directed learning groups differed across all 
genres of production, here we found that the higher creative production group had a higher depth of 
experience than the lower (M=3.32, SE=0.36; M=2.13, SE=0.37; F(l,46)=5.34, p=.025) and for expressive 
activities (M=2.36, SE=0.19; M=1.35, SE=0.20; F(l,46)=13.14, p=.001), and the higher social learning 
group had higher depth of experience than the lower (M=3.64, SE=0.36; M=2.00, SE=0.33; 
F(l,46)=11.20, p=.002) and for both expressive (M=2.36, SE=0.21; M=1.46, SE=0.20; F(l,46)=9.79, 
p=.003) and publication activities (M=0.55, SE=0.12; M=0.19, SE=0.11; F(l,46)=13.14, p=041). This 
suggests a portrait of all of Kellog's students who were active online as deeply engaged in sustained 
media production, with those in the higher creative production group focusing on expressive media, 
those in the higher social learning group focusing on expressive media and publication as they share 
their ideas with others, and the self-directed group engaged in sustained participation across gen res of 
production, including computational. 

Importantly and obviously, there are serious I imitations to this approach and interpretation of results. 
Because this data is correlational we do not attempt to make causal claims nor do we claim that actions 
in the system are equated with an instance of learning, but instead report on relationships between 
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experiences of opportunity logged in the system and self-report of experiences and perceptions. We do 
not know if the students who are coming in with more evidence of 21st century learning are usingthe 
site in different ways or vice versa, nor can we say that a click on the system that logs an activity is equal 
to a learning activity. Rather we look at these logged activities as opportunities for learning. This 
research was emergent, and we used data collected over a six month design-research effort, including 
surveys and interviews, to better understand the iRemix log data that was collected. Also while we 
report statistical significance and use the accepted level of less than 5% (.05) to determine that 
differences are not happening by chance, we recognize that for each of the findings there is the 
potential foratype I error, meaninga null hypothesis istrue. 

5.4 Focal Learner Cases: Amaya and Sterling 

To better understand the results in the online log data and the triangulation of the log patterns with 
surveydata, wedevelopedqualitativecasesofindividual learners. This allowed us to learn about the 
backgrounds they brought to the experience of using iRemix in the classroom and more about their own 
intentions of use. In the sections below, we share narrative stories of two of Kel log's students: Amaya, 
who scored in the higher group for all three types of 21st century learning activity on iRemix, and 
Sterling, who scored in the lower group for all three (see Figure 7). Amaya had a creative production 
opportunities count of 111 (89th percentile), a self-directed learning opportunities count of 35 (79th 
percentile), and a social learning opportunities count of 74 (69th percentile). Sterling had a creative 
production opportunities count of 40 (6th percentile), a self-directed learning opportunities count of 22 
(19th percentile), and asocial learning opportunities count of 61 (50th percentile). 



Figure 7: Histograms of 21st century activity counts with focal case learners located, a light circular 
marker for Amaya and a dark circular marker for Sterling. 
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5.4.1 Amaya: embracing storytelling through digital media and making plans for learning more 

I usually I ike to think outside the box. So I I ike to be unique with my [digital media] stuff, like not 

the same as everyone el se's, I ike at least one thing that stands out. 

Amaya was a 12-year-old girl who described herself as Latina. She lived near the school with her parents 
and her younger brother. She was involved in a number of regular extracurricular activities, including 
cheerleading, karate, piano, and guitar lessons. Amaya was highly engaged in narrative stories of any 
kind, and enjoyed watching movies and reading books in her free time. She had personal access to 
technologyandthe Internetathome, includingawirelessconnectionand herown MacBook, and was 
very interested in technology. On he ri Remix profile, she declared her affinity with computers, "When it 
comes to computers I go crazy! I could spent hours on them, but then have no ti me to read!" Her entire 
family regularly used computers for basic Internet searches and entertainment, and her mother, a 
daycare teacher, used it for work to create simple lesson plans. On her own time, Amaya frequently 
went online to search for information about her hobbies, use social networking sites, and I ess frequently 
played adventure games. She specifically wanted to learn more about "howto make websites, and more 
video editing." 

Her interests and pursuits outside of iRemix very much align with her counts of online learning 
opportunitiesthatplace herinthe highergroupsintermsof all three dispositionsthe DYN program is 
interested in fostering — creative production, self-directed learning, and social learning. 

Creative Production. Amaya logged one of the highest counts of potential creative production 
opportunities in iRemix in her grade. Offline, she similarly identified very much as a creative producer, 
both in and out of school. She was proud of the work she created and found ways to blend he r growing 
technological knowledgewith herpersonal interests. She described usingtoolson herMac at home to 
create and edit videos and images, including using Pixlr, a tool learned in Kel log's class, to edita photo 
her father had taken of her doing karate, to make it more dramatic and "really cool." In school, her 
interest in narratives in different formats immediately tied into Kellog's class focus on multiple modes of 
expression. 

Her favourite school project was a book report in Kel log's class where students were given the option to 
create an imagined video movie trailer for the story. "I spent a lot of time with it," she related, "and it 
was more unique than the rest of them because it was in first person view and everyone else did it in 
third person view. I showed it to my parents. They thought it was cool."The book chosen by Amaya was 
Cinder, a popular young adult novel retelling the Cinderella fairy-tale as dystopic science fiction (see 
Figure 8). Amaya blended film footage, still photos, sound effects, layered musical soundtracks, and 
captions with subtle transitions into a two-and-a-half minute movie that conveys a mo vi ng portrait of 
futuristic and social hardship from the perspective of the protagonist. Amaya played on traditional 
movie trailers (using opening shots from Time Warner), the book's genre bending (the traditional "Once 
upon a time" shot is foil owed by the sound of a scratching record and the words, "I wish I could tell you 
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that story, but it's not mine"), and uses elements of popular Sci-Fi and horror movies (escalating music, a 
blank screen and pause in audio, and a disturbing phone-call conveyed in words and images). 



Figure 8: Stillsfrom Amaya's book reviewtrailer digital movie. 


Amaya described her use of iRemix as a combination of posting assigned work and sharing her own 
creations and other found media. During the six months of log data collection, Amaya posted 14 blog 
posts, 13 videos, 8 images, and 2 audio files to the site, and had two notes in her notebook. Like her 
video book report submission, many of her posts were related to contemporary young adult fiction, as 
whenshe shares herperspectiveson bookslike The FaultinOurStars andThe FlungerGames. 

Amaya's role as a creative producer of digital media was recognized through her work in Kel log's class. 
Kel log reflected that Amaya felt more comfortable sharing her talents and ideas through movies, "She's 
really quiet, but she'ssupercreativewith heriMoviesand just really outgoing." When asked which of 
their peers was good with computers, Amaya was the seventh most nominated in her grade (13% of 
students selected her). Positive comments on herworkwere frequently posted by herpeersin iRemix, 
such as this one on her book movie trailer: "Amaya, this is sooooooooooooooo cool I love the end!" 

Self-Directed Learning. Aligning with herhigh countof self-directed learningopportunities in relation to 
herclassmates, Amaya revealed a numberof ways in which she regularly navigated herown learning as 
she connected to networked tools and information online. For example, although she was unsure about 
the availability of video editing classes in high school, she planned to "continue to teach myself. I am 
mostly using the Internet," and talked specifically about foil owing individuals who post tutorials about 
video editing on sites like YouTube. She also engaged in online research to troubleshoot problems that 
came up when she worked through digital media projects or other homework, and used Kahn Academy 
on herown time to play educational math games. This self-direction is evident on her iRemix portfol io, 
which documents final versions of all assigned work and completion on all assigned learning pathways 
despite the fact that this level of closure was not necessarily req uired by Mr. Kel log for the digital media 
projects in his class. Amaya recognized the iRemix portfolio space as a way to keep track of work 
(students often used different laptops from day to day) and to showcase it outside of class. She 
reported, "iRemix is an easy way to know where you put a certain thing," and talked about using the 
system to show her media submissions to friends and family at home. Amaya logged onto iRemix 
outside of cl ass time and described posts from home to "finish something up." She also planned to use it 
in the upcoming summer "to keep my brain fresh ... to get extra practice." She wanted more learning 
pathways tied to her specific interests, specifically cove ring more advanced video editing. 
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Social Learning . Although Amaya was both teaching and learning from others about computers and 
technology at home and at school, the social dimension was secondary to her independent work, and 
when she was social online it was often specifically tied to media and production. This aligns with her 
count of social learning opportunities online, where Amaya positioned her closer to the middle in 
relation to her peers, although she still remained in the higher group. 

Amaya shared her knowledge with her mother, including helping her to use Google drive to create 
graphs, and at school 18% of her grade peers selected her as someone who they learn from. On i Re mix, 
she engaged in the community aspect of the system by creating and joining iRemix groups tied to her 
interests in movies and popular novels. She enjoyed sometimes commenting on her friends' work and 
took pride when she received comments on her own work, "It makes me happy that they took the time 
out of their day to look at my work. Yeah because they're like, 'This is cool, no one else posted this, let's 
check this out'." Her main social learning activity was to look at the work of others on iRemix, and she 
talked about learningfrom observation, "I liketoseeotherpeople'screations, like I mightget inspired 
and I have a couple of times by other people." Amaya's comments to others on the system were 
specificallyaboutelementsoftheirwork, "I like how the music matched the tone of your poem! Even 
though it was sad, I really liked it." 

5.4.2 Sterling: focusing on the performing arts and figuring out how to support it with production 

I love performing ... A lot of people think I'd be good on Broadway and stuff. 

Sterling was a twelve-year-old African American male actively involved with the performing arts, who 
described himself as an actor, singer, and dancer. He lived with his parents, two older brothers, and two 
older sisters. He participated in an afterschool choir program at a city high school but attended 
infrequently because it was difficult to find transportation consistently. Although he received good 
grades in school, he described himself as a "slow reader." He had his own computer at home, and the 
family had another computer, a tablet, and an Internet connection that Sterling described as slow. He 
regularly used the computer to playgames, listen to music, and engage in social media sites, including 
Twitter and Face book, at the library after school or at home after he finished homework. 

His interests and pursuits outside of iRemix somewhat align with counts of online learning opportunities 
that place him in the lowerof each of the creative production, self-directed learning, and social learning 
groups, but his qualitative case portrait helps to point out important nuances to supplement the story 
told by the log data. 

Creative Production . Despite the fact that he was considered tech savvy at home, he did not define 
himself as a creative producer nor was that how he was perceived at school. When describing his iRemix 
profile page, he explained his focus on exercising creativity and expression through performance, "[In 
'About Me'] I wrote 'Next on stage, he is very talented! He is such an amazing singer!'This is stuff that I 
want to do when I grow, like be a singer or whatever. This is how I wanted to express myself." Although 
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he saw the potential foriRemixasan outlet for performance, specific activities to do so were not clear 
to him and they were not part of any of the existing learning pathways on the system, "If you are doing 
something like music-related then yeah. It depends on what subject and what you are doing in what 
subject." 

The lack of obvious connections with his out-of-school interests resulted in iRemix being primarily a 
placeforteacher-initiatedassignmentsforSterling. Duringourstudy, he posted3images, 2 sounds, 3 
videos, 3 blog-posts, and 3 note book entries; all required assignments. None of his peers identified him 
as someone who was knowledgeableabout computers noras someone they learned from. In terms of 
his own perception of himself as a creative producer and public recognition in this area at school. 
Sterling's low creative production count is in alignment. At home, however, he dabbled in his own movie 
making. He transferred what he learned in iMovie on the school MacBooks in Kellog's class to his Dell 
computer, "It has these apps where you can make your own movie, but it's not called, like on Apple 
computer, iMovie, it's just cal led My Movie and Movie Creator, and it'ssorta like iMovie, but they have 
more effects and more stuff that you can put on there." He also had ideas for how to support his passion 
for the performing arts through digital production, such as making music videos of his performances or 
creating songs on digital audio editing programs, but felt he did not yet have these skills to dothis and 
did not see these activities as tied to his work in Mr. Kellog's class or on iRemix. 

Self-Directed Learning . Aligning with his lower self-directed learning opportunity scores on iRemix, 
Sterling did not pursue these digital production I earning goal son his own and was sometimes frustrated 
when working independently. Sterling used the Internet to seek out potential future learning 
opportunities related to his interests, such as searching for potential performing arts high schools and 
colleges, but was notable to articulate a plan for his more immediate and direct desire to learn to create 
websites and produce quality audio and video pieces. He also described feeling stressed when he faced 
technical issues while working with technology, "Sometimes it gets difficult. ... Sometimes the 
computers have its own mind. It takes hard work. I also get stressed a lot because if what you searched 
for is not on there, then you get frustrated and stuff." On iRemix, Sterling discussed sometimes going on 
outside of class time, but it was not tied to specific learning goals, "Once I was bored and it was a 
weekend and I didn't have nothing else to do so I went on to find extra credit work." Sterling did not 
complete any of the learning pathways that involved more than one activity step, suggesting that he had 
trouble persisting through multi-step projects: he completed one of the four Welcome to iRemix 
challenges, and two of the four RACE challenges. 

Social Learning . While in the lower group, Sterling's count of opportunities for social learning on iRemix 
was close to Amaya's and close to the middle in comparison to his classmates. Sterling was a very social 
person who enjoyed the company of others and sought an audience. The social nature of iRemix was 
what most easily fit with Sterling's interests and intentions online. At home, when his family needed 
help with the computer, they would ask Sterling or his 15-year-old sister, and he taught his brothers how 
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to crop images and make presentations in PowerPoint. He became engaged in social media after hearing 
other students talk about their followers, "I decided to do it/cause it was making me so competitive ... 
also not just for that, I just wanted really just wanted to socialize with my friends and family that was far 
away from me." 

The best part of iRemix for Sterling was the opportunity to share his work and profile to connect with 
others. He frequently used iRemix to show his family the work he was doing in class. He once showed his 
mother a blog he wrote about a family vacation and was happy that she liked it and that "it brought her 
memories when she was reading it." He especially enjoyed receiving positive feed back on his posts, but 
not necessarily those that might lead to revisions, "[I] hope for good comments, less stuff [I] need to 
improve on." His comments on other people's work were always encouraging, "I love your eat healthy 
song sis, KEEP IT UP!" Sterling, in fact, wished that iRemix social functionality was more person-to- 
person oriented instead of focusingon communication around submitted artefacts, "In othe r we bsites 
you can leave messages and have conversations but you can't have it there. You can leave comments 
and stuff, but if there is a comment box with just that person then it would much easier." He used the 
groups feature to connect with others more around his general interests than artefacts, including 
joining, becominga coordinator of, and postingto a group called "Beyonce Fans" with 13 members, and 
creatinga group based on love of the colour purple, where he was the sole member. Whilesterling was 
not interacting around work and formal ideas as much as Amaya, these community and social 
opportunities logged around his personal interests identify the way he was able to engage with the 
learningcommunity. 

5.4.3 Case learning comparison 

The 21st century log data framework positions Sterling and Amaya in terms of dimensions of creative 
production, self-directed learning, and social learning on iRemix. The way each student's log traces 
classified them make general sense: Amaya was a creative producerof digital media who both identified 
with and was recognized for this role, and regularly engaged in independent learning practices, and 
social interactions related to media production. Sterling was uncertain how to I ink digital production to 
his personal interests and found it difficult to navigate his own learning, although he coordinated a 
social sphere of learning partners for community interactions. While this was encouraging, the 
qualitative case portraits allowed us to see nuances both in what was happening and for what was 
possible. Both Sterling and Amaya considered themselves to be very creative, and this was an important 
part of their personal identity. Amaya was a creative producerof digital media such as video and artistic 
projects, easily tying her personal interests in narratives to her production efforts, and this was evident 
on iRemix. Sterling's sense of his creativity was focused on performance and as of yet, he had not 
translated those pursuits into digital production. As such, Sterling did not have a strong reputation for 
creative production in the classroom nor did he log high counts of opportunities on iRemix, although his 
case reveals ample opportunity for encouraging a learner like him to do so. In terms of se If-directed 
learning, Amaya had clear learning goals and plans for achieving them, including a solid command of 
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utilizing networked resources on the Internet and a history of completing the work she began in i Re mix 
learning pathways, while Sterling did not see opportunities for learning on his own and sometimes 
became frustrated trying to do so. These representations correlate to the i Re mix logs, where Amaya was 
very high compared to herclassmatesandSterlingwasonthe lowerend. The social learning grou p was 
where these students look similar. Although Amaya was in the highergroup and Sterling in the lower 
group, their actual counts were very close; both enjoyed the social learning dimension of the site. 
Amaya was more likely to discuss social learning activity as a means for learning more, whereas Sterling 
highly enjoyed the social aspect of this type of learning activity, which we know is still important for 
learning as he builds community online, with the potential for future recognition oralignmentaround 
interests or expertise. Although Amaya was recognized by classmates as having expertise in digital 
production through sharing her work and teaching others, aligning with her high counts on the public- 
facing iRemix site, both used iRemix to showcase work at home with their parents, and both taught 
family members and were called upon for help with computers and technology at home. 

6 DISCUSSION 

In this paper, we present a multi-level approach to understand how log data might reveal patterns of 
activity that representopportunitiesforcreative production, self-directed learning, and social learning. 
This study was motivated by the increasing presence of K-12 online learning environments and concern 
about equitable access and generative participation for all youth. We posit that the data that can be 
collected from such systems offer enormous potential to understand how and when youth take up 
opportunities for 21st century learning practices and how best to support it. While our work is 
exploratory, in the following section, we highlight some of the interestingfindings, implications, and 
opportunities for further study that have emerged in the course of this study. 

6.1 Interpretation of Logged Actions: Actor and Recipient 

To interpret the user traces, we looked at logged activities in terms of the actor and the recipient of 
each activity, attending to roles of self, peer, and educator, aligning with approaches defined in social 
learning analytics (Buckingham Shum & Ferguson, 2012). We then organized a conceptual framework of 
logged actions building on prior DYN design-based research and theoretical constructs from recent work 
in the fields of learning sciences and human-computer interaction. Although this work is still in early 
stages, we found patterns of user trace data logs that triangulated with self-report measures of 21st 
century learning dispositions that DYN was designed to foster. Specifically, students with more creative 
production activity oniRemix were more likely to identify themselves as pursuing possible futures as 
engineers,computer-enabled designers,andvideoproducers,suggestingpersonal identification as a 
technological creator. Those with more independent I earning activity on iRemix had higher measures of 
sustained creative project experience across different genres of media, including computational 
activities such as programming and robotics, suggesting possible connections to persistence, use of 
personal and material resources, and troubleshooting. Those students who evidenced more social 
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learning activity on iRemix were more likely to agree that they got new perspectives on social issues 
through their use of the Internet, and reported learning from fewer people in the face-to-face contexts. 

There were no differences of logged activities in the three 21st century learning themes we looked at i n 
terms of student home access, which we use as a proxy for socio-economic status, indicating similar 
usage of iRemix by students from households with varying learning supports. Gender did not play a role 
in creative production orself-directed learningopportunities on iRemix, although girls engaged in more 
social learning opportunities logged on the system. Although the boy and girls in this study reported 
similar frequency of use on other social network sites, they were all under 13 years old, meaning that 
they were not technically allowed on public social media platforms as per COPPA standards — these 
girls perhaps had not yet begun to differentiate in more publicsocial networks. National studies of teen 
use of technology platforms have found that girls dominate some social media platforms, especially 
those revolving around visual artefacts (Lenhart, 2015). Looking for ways to ensure that boys remain 
engaged in social learning activities around media artefacts in iRemix as they develop as young adults is 
an important outcome of these findings. 

We believe that ways to look dee ply at activities online that go beyond more traditional metrics of time 
on task and completion rates are critical to unde rstan ding systems for different populations, and believe 
that the method of coupling action with the relationship between actor and recipient is an important 
contribution. Our next step is to carry out a systematic research intervention to validate our exploratory 
conceptual coding frame works using external measures of 21st century learning, such as existing verba I 
and performance-based measures of creativity (e.g., Silvia et al., 2008) and persistence (e.g., Shute & 
Ventura, 2013). 

6.2 Interpretations of Logged Actions: Deep Knowledge of the Environment 

Each context of use determines success differently based on intentions, and the system should allow for 
multiple representations and visualizations of data to reflect different potential uses of the system, 
attending not only to more traditional metrics of time on task and completion rates, but also to 
practices that may be evidence of opportunities to develop important skills and dispositions that go 
across specific content areas and grade levels. Learning deeply about our focal learning environment, 
including the community and the teacher and his students, allowed us to further interpret the results 
that emerged in our coding of the logged user trace data. In other words, quantitative data begins to tell 
the story, but to design better environments and practices within those environments, especially in 
ways that can ensure that all youth have access to generative learning opportunities in online 
environments, we need other strategies to piece together a more complete picture of what is going on. 

Although iRemix was designed with intentional support for student-driven activity, it purposefully 
includes learning management features for adults in the system, to guide use. In Ke I log's classroom, his 
intentionality as a teacher and the climate of the I earning environment influenced student use of the 
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site. The majority of logged activity was related to creative production, aligning with Kellog's strong 
focus on fluency and personal expression with digital media production. This was connected to his 
introduction to the system through DYN and the digital literacy initiative in his classroom during his first 
year of teaching. It was also connected to ideas about his student population, including wanting to 
increase their ability to express themselves in multiple modalities given their reluctance for sharing in 
person. For Kellog, successful use of the system was evidence of generative student creative production 
on the site, including assigned and voluntary posts and social interactions around those posts. 
Importantly, two key factors that made it possible for him to use this platform in a way he deemed 
successful were long-term regular use of the system with his students — all owing for the establishment 
of community norms and a growing body of submitted work and activity — and the support of his 
administration to implement an experimental system used for more free expression in a core subject 
classroom. 

We are very interested in the way Kellog used informal exploration of digital media to intentionally 
encourage youth voice in his classroom, which has interesting implications for using an online platform 
to promote online contributions from under-represented populations. Studies have revealed that 
contributors of online content in general are a small subset of the population usingtechnical systems, 
and that this subset is not representative of the larger population (e.g., Glott, Schmidt, & Ghosh, 2010). 
As more people select online sources, including social networking sites, for social and political 
information, the re are implications related to the critical need for equitable and balanced perspectives 
on the Internet. 

The importance of exposure to and experiences in participatory cultures and community for building 
skills, developing interests, and becoming creators and contributors is well documented (Hargittai & 
Walejko, 2008; Ito et al., 2009; Jenkins et al., 2009). Log data showed many opportunities for creative 
production and social learning, but not as much for self-direction. Research in higher education has 
found that while some students are able to utilize social I earning networks like i Re mix to drive their own 
learning needs, importantly, not all students possess the knowledge management or self-regulatory 
skills to do so, and authors point out the need for scaffolded instructor use of system features (Dabbagh 
& Kitsantas, 2012). Consistent with this, our study documented wide variability across individual youth 
learners on the system. Previous research has identified characteristics of lifelong learners as 
responsible for their learning process, able to span formal and informal, and heterogeneous in their 
approach to I earning goals that they look for and achieve (Koper& Tattersal I, 2004). 

We saw that while Amaya was a I earner who regulated her activities, Sterling needed outside gui da nee 
and was more likely to stay within the boundaries of what was asked of him by the teacher. These case 
portraits enabled us to see nuances within general alignments and, as such, opportun ities for better 
design, practices, and teacher professional development to connect youth to more engagement in 
potential learning opportunities online. For Amaya, the content of the class using multiple media to 
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explore traditional literacies mapped onto her existing interests, and this led her to pursue more 
resources, look at the work of peers, and find opportunities to learn more. For Sterling, the connection 
between the course content and his personal interests and identity was not as obvious, but he was 
slowly finding ways to explore his own interests in iRemix through his profile page and by organizing 
special-interest communities, like his Beyonce group. Opportunities to engage Sterling in media 
production around his performance artistry and creativity perhaps could become more possible for the 
teacher and peers to recognize as Sterling became more visible as a creative person within the i Remix 
site. Both case learners were interested in learning more digital production tools, including web and 
video development; opportunities for embedding or I inking resources and visible pathways to support 
youth to go further in their learning outside of the boundaries of the classroom is something we are very 
interested in implementing. 

6.3 Translating Generative Learning Intentions 

Educators and administrators need shared language to better articulate how their learning goals map 
onto potential outcomesand practiceswithinonlinesocial learningnetworkssotheycan fully use the 
system and support their learners in ways that they care about, and so that they can influence 
purchasing decisions at the school and district levels. In our study, Kel log embraced the use of iRemix for 
creative production, a practice he established during his first year of teaching. While iRemix included 
features for other types of learning opportunities that aligned with his ELA curriculum, such as 
discussion forums and debates, Kellog "stuck with what he knew." Although he spoke of consciously 
trying to support his students to take control of their own learning, system log data that we associated 
with self-directed learning opportunities on iRemix indicates that these features were almost never used 
in his classroom, such as monitoring progress, using embedded resources, and seeking help for 
assignments. For educators, there is a demand for professional development that not only helps them 
use technology, but also helps keep them "technologically agile" in a world of rapid technological 
change (Borgman et al., 2008; ACCI NSF, 2011). Recent reports on the use of online learning systems 
suggest that schools often do not realize the training, time, and effort necessary to establish new 
systems in the classroom (Watson, Murin, Vashaw, Gemin, & Rapp, 2011). While ongoing useful 
professional develop mentis critical, it cannot keep up with the rate at which new learning technologies 
and platforms are being introduced at district, school, and individual classroom levels. We posit that 
clearly articulating how features and corresponding student and educator use of those features map 
onto opportunities for learning may be one way to help teachers navigate environments for their 
classroom in more intentional ways. Organizing our features and interpretations of logged activity traces 
into themes aligned with DYN goals and national efforts to define 21st century learning allowed us to 
talk to educators about both the site features and capabilities, and potential opportunities for youth 
related to those aspects of the system. We believe that continuing to develop and use language for 
looking at evidence of 21st century I earning online is critical for translating what we see. Teachers need 
to know how to translate intentions that may be more abstract, such as Mr. Kellog wantingto nurture 
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self-directed learners, into practical I earning designs and online tasks, selecting appropriate tools and 
features. We see this as an important step in supporting practitioners to use platforms for more than 
just traditional distribution of assignments and quizzes and collection of homework, and moving forward 
to help design better online learning environments across different content, ages, and communities. The 
abilitytologactivitiesthatgo beyond time-on-task and submissions makethesemore abstract types of 
learning actions and opportunities more assessable for teachers in both formative and summative ways. 

6.4 Using a Bottom-Up Approach 

This paper presents an exploration of a socio-technical system, specifically looking for interesting 
patterns in quantitative logged usertracesand tryingto understand the patterns more deeply through 
self-report data and qualitative cases. Although it is not the focus of this paper, this work is part of a 
larger three-year design-based research project (Collins, 1992; Brown, 1992; Hoadley, 2002) to 
understand learning environments in real-world contexts using a process of design, implementation, and 
iterative refinement. 

The DYN organization is comprised of educators, learning scientists, and computer scientists. To make 
sense of the data generated from logged activity over three years — including engaging in powerful 
learning analytics and educational data mining to look at clusters of activities, change overtime, and 
profiles of users — we realized we first needed a crystal cl ear understanding of the i Re mix system that 
had emerged over the course often years, to the detail of each minute of activity that was logged. To 
engage in cycles of design, implementation, and iterative refinement, and utilize these new processes of 
data analytics to reflect learning, we needed a model that 1) our entire team felt was authentic to the 
DYN program and design intentions, 2) reflected the way the system was used in different contexts, and 
3) practitioners could also understand. The methodology and framework of 21st century learning 
opportunities takes into account the system features, the data it collects, the actors in the system, and 
the objects they interact around. As such, we developed these higher order constructs related to 
opportunities on the site forcreative production, self-directed learning, and social learning, grounded in 
lower level activity logs. This allows us to attend to practices of use (e.g., Which features are being used 
and which are not? What opportunities for I earning are not being taken up by educators or youth on the 
system? How can we better support youth?), and to make sure we understand individual actions that 
are logged on the system, and that we are collecting the right actions with enough meta -data to make 
interesting claims about learning. Without this deep understanding, alignment, and agreement, the data 
generated from more top-down approaches did not make sense to the team of practitioners and 
learning designers, and did not, therefore, result in direct opportunities for action, i n cl u ding platform 
redesign and revisions to professional development to promote teacher practices. 
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7 CONCLUSION AND FUTURE WORK 

The more we can define, cluster, and compare different types of environments, the easier it will be to 
conduct meaningful research, interpret results, and build on the research efforts of others. The potential 
for mining online data and applying learning analytics is exciting, but we need to be clear about different 
types of online environments, their various features, their learning goals, and potential outcomes to be 
able to design robust research efforts and analyze and begin to compare results. We present our 
methodology of automated coding of 21st century learning opportunities as part of a larger initiative to 
understand and discuss the rapidly growing number of online I earning environments, and we believe it 
to be both beneficial and necessary for designers, practitioners, and researchers in and of such 
environments. 

Our coding methodology is intended to illuminate the kinds of moves youth made online including 
youth-to-system interaction and youth-to-person interaction using automated coding of user trace data 
logs, with specific attention to relationship types of log action, actor, and recipient. Building fro m the 
foundational learning ideas that support DYN design and programming, we developed a coding 
frame work to look at opportunities for 21st century I earning opportunities on i Re mix within themes of 
creative production, self-directed learning, and social learning. We intendtousethisframeworktolook 
for changes in activity across time, and to develop visualizations to use with practitioners in ways that 
allow them to compare their learning goals with online activity, and see opportunities for changes in 
classroom practice and use of tools to promote different types of outcomes. We are also using this 
methodology to look at the system use in different environments with unique learning goals and 
populations, to find out more about how different practices determine student activity overtime and 
what success looks like across contexts. In this paper, we have documented how we explore the 
authenticity of our logged data through student self-report survey data and qualitative case studies of 
individuals in the learning environment. We plan next to engage in empirical work to see if logged 
opportunities for 21st century learning correlate with validated external measures of 21st century skills. 

Although i Re mix does not adhere to current learning technologies' language standards, the user trace 
coding methodology shared in this paper may be of interest to the learning analytics community and has 
the potential to be adapted for use with data from other platforms. For educators and professional 
development leaders, this work may also advance ways to articulate and share learning goals, 
opportunities, and patterns of use of online environments related to 21st century learning, aiding 
discussion and understanding of the complexities of educator roles and student behavi ours online. 
Above all, our work is an exploration of and ongoing design effort to create online learning 
environments as spaces that can provide youth with digital learning resources and a knowledgeable 
social network to support learning and development. 
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